Skip to content

Commit 9493df7

Browse files
Merge branch 'main' into 157
2 parents f4d4dfc + 29db12e commit 9493df7

9 files changed

Lines changed: 652 additions & 14 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ Latest
77
* [#143](https://github.com/cleverage/process-bundle/issues/143) Improve PHPStan level from 6 to 7, and fix the level 8 errors that do not require a signature change (the remaining ones, due to nullable return types such as `AbstractConfigurableTask::getOptions(): ?array`, will be fixed in v6.0). Default to a `ContextualOptionResolver` in ProcessState and to a property accessor in ConditionTrait when none is set.
88
* [#94](https://github.com/cleverage/process-bundle/issues/94) TransformerTrait: the `transformers` option (TransformerTask, MappingTransformer, ArrayMapTransformer, CachedTransformer, RulesTransformer, generic transformers) also accepts a list, whose items are a transformer code without options (`- trim`) or a single `code: options` map (`- callback: {...}`), to chain the same transformer without `#` suffix. The map syntax is still supported. Update documentation, add tests.
99
* [#117](https://github.com/cleverage/process-bundle/issues/117) MappingTransformer: writing a target property that is a nested path (e.g. `address.city`) on an array destination, which silently created an `address.city` literal key, is deprecated and will throw an `\UnexpectedValueException` in v6.0. Use `[address.city]` for a literal key or `[address][city]` for a nested array; a simple property name (e.g. `name`) is still added as a key. Update documentation, add tests.
10+
* [#258](https://github.com/cleverage/process-bundle/issues/258) Add XmlStreamReaderTask: iterable task reading an XML file element by element with `\XMLReader` (constant memory), outputting each element named `node_name` as a `\DOMElement` in its own `\DOMDocument`, ready for XpathEvaluatorTransformer. The file path is set in the `file_path` option or given as input. Add documentation and tests.
11+
* [#259](https://github.com/cleverage/process-bundle/issues/259) JsonStreamWriterTask: add a `mode` option (default `wb`, e.g. `ab` to append to an existing JSON Lines file), and release the file on `finalize()`. Update documentation, add tests.
1012
* [#157](https://github.com/cleverage/process-bundle/issues/157) Configure the level of the logs written at the end of each process with the new `logs` option, globally (`clever_age_process.logs`) or per process (`configurations.<code>.logs`): `success_level` (default `info`) for `Process <code> succeed`, and `failed_level` (default `debug`) for the new `Process <code> failed` log, written when a process fails, with or without exception. Update documentation, add tests.
1113

1214
## Fixes

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@
5050
- [JsonStreamWriterTask](reference/tasks/json_stream_writer_task.md)
5151
- File/XML
5252
- [XmlReaderTask](reference/tasks/xml_reader_task.md)
53+
- [XmlStreamReaderTask](reference/tasks/xml_stream_reader_task.md)
5354
- [XmlWriterTask](reference/tasks/xml_writer_task.md)
5455
- File/Yaml
5556
- [YamlReaderTask](reference/tasks/yaml_reader_task.md)

‎docs/reference/tasks/json_stream_writer_task.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ Options
2626

2727
| Code | Type | Required | Default | Description |
2828
|-------------------------|---------------|:--------:|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
29-
| `file_path` | `string` | **X** | | Path of the file to write to, opened in `wb` mode (overwritten); missing parent directories are created.<br/>Placeholders `{date}`, `{date_time}`, `{timestamp}` and `{unique_token}` are replaced in the path |
29+
| `file_path` | `string` | **X** | | Path of the file to write to; missing parent directories are created.<br/>Placeholders `{date}`, `{date_time}`, `{timestamp}` and `{unique_token}` are replaced in the path |
30+
| `mode` | `string` | | `wb` | File open mode (see [fopen mode parameter](https://www.php.net/manual/en/function.fopen.php)): `wb` overwrites the file, `ab` appends to it |
3031
| `spl_file_object_flags` | `array\|null` | | `null` | List of `SplFileObject` flags, summed and passed to [SplFileObject::setFlags](https://www.php.net/manual/en/splfileobject.setflags.php).<br/>`null` means `DROP_NEW_LINE`, `READ_AHEAD` and `SKIP_EMPTY`; an empty array means no flag |
3132
| `json_flags` | `array\|null` | | `null` | List of JSON flags, summed and passed to [json_encode](https://www.php.net/manual/en/function.json-encode.php).<br/>`null` means `JSON_THROW_ON_ERROR`; an empty array means no flag |
3233

@@ -62,3 +63,4 @@ Notes
6263
* The parent directory of the file is created (recursively) if it does not exist.
6364
* Using `JSON_PRETTY_PRINT` produces multi-line documents: the file can no longer be read by
6465
[JsonStreamReaderTask](json_stream_reader_task.md).
66+
* The file is released at the end of the process.

‎docs/reference/tasks/xml_reader_task.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,5 @@ Notes
4747
* An invalid XML content (not well-formed, undefined namespace prefix...) throws an `\UnexpectedValueException` whose
4848
message contains the libxml error(s) with their line and column. libxml warnings are tolerated. The libxml internal
4949
errors setting is restored after loading.
50+
* The whole file is loaded in memory. Use [XmlStreamReaderTask](xml_stream_reader_task.md) to iterate over the elements
51+
of a large file.
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
XmlStreamReaderTask
2+
===================
3+
4+
Reads an XML file element by element with [XMLReader](https://www.php.net/manual/en/class.xmlreader.php) and iterates
5+
over the elements with a given name, outputting each of them as a `\DOMElement`. Unlike
6+
[XmlReaderTask](xml_reader_task.md), the whole file is never loaded in memory: use it for large XML files (product
7+
catalogs, ERP exports...).
8+
Requires the `dom` and `xmlreader` PHP extensions.
9+
10+
Task reference
11+
--------------
12+
13+
* **Service**: `CleverAge\ProcessBundle\Task\File\Xml\XmlStreamReaderTask`
14+
* **Iterable task**
15+
16+
Accepted inputs
17+
---------------
18+
19+
`string`: path of the file to read, only used when the `file_path` option is not set (ignored otherwise).
20+
21+
Possible outputs
22+
----------------
23+
24+
`\DOMElement`: for each element named `node_name`, the element expanded with
25+
[XMLReader::expand](https://www.php.net/manual/en/xmlreader.expand.php), as the root element of its own
26+
`\DOMDocument`. It can be passed directly to the [XpathEvaluatorTransformer](../transformers/xpath_evaluator_transformer.md), with
27+
queries relative to the element (e.g. `@id`, `name/text()`).
28+
29+
When the file contains no matching element, no output is produced and the task is skipped.
30+
31+
Options
32+
-------
33+
34+
| Code | Type | Required | Default | Description |
35+
|-------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------------------------|
36+
| `file_path` | `string\|null` | | `null` | Path of the file to read. When `null`, the path is read from the input |
37+
| `node_name` | `string` | **X** | | Qualified name of the elements to iterate on, as written in the file (e.g. `product`, or `g:item` for an element with a prefix) |
38+
39+
Examples
40+
--------
41+
42+
```yaml
43+
# Task configuration level
44+
read:
45+
service: '@CleverAge\ProcessBundle\Task\File\Xml\XmlStreamReaderTask'
46+
options:
47+
file_path: '%kernel.project_dir%/var/data/products.xml'
48+
node_name: product
49+
outputs: [transform]
50+
transform:
51+
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
52+
options:
53+
transformers:
54+
xpath_evaluator:
55+
query:
56+
sku: '@sku'
57+
name: 'name/text()'
58+
outputs: [write]
59+
```
60+
61+
Notes
62+
-----
63+
64+
* Elements are matched at any depth. The subtree of a matching element is not searched again: an element named
65+
`node_name` nested inside a matching element is part of its parent output, and is not output on its own.
66+
* Namespaces declared on ancestors are copied to the expanded element, so prefixed queries (e.g. `g:price/text()`)
67+
still work.
68+
* An invalid XML content (not well-formed, empty file...) throws an `\UnexpectedValueException` whose message contains
69+
the libxml error(s) with their line and column, handled by the task `error_strategy`. As the file is read as a
70+
stream, the elements located before the error have already been output. libxml warnings are tolerated.
71+
* A missing or unreadable file, or no file path in the option nor in the input, throws an `\UnexpectedValueException`.
72+
* The file is released at the end of the iteration (and on `finalize()`), so that the task can read it again on its next
73+
input.
74+
* Use [XmlReaderTask](xml_reader_task.md) to get the whole document at once (small files).

‎src/Task/File/JsonStream/JsonStreamWriterTask.php‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,12 @@
1616
use CleverAge\ProcessBundle\Filesystem\JsonStreamFile;
1717
use CleverAge\ProcessBundle\Model\AbstractConfigurableTask;
1818
use CleverAge\ProcessBundle\Model\BlockingTaskInterface;
19+
use CleverAge\ProcessBundle\Model\FinalizableTaskInterface;
1920
use CleverAge\ProcessBundle\Model\ProcessState;
2021
use Symfony\Component\OptionsResolver\Options;
2122
use Symfony\Component\OptionsResolver\OptionsResolver;
2223

23-
class JsonStreamWriterTask extends AbstractConfigurableTask implements BlockingTaskInterface
24+
class JsonStreamWriterTask extends AbstractConfigurableTask implements BlockingTaskInterface, FinalizableTaskInterface
2425
{
2526
protected ?JsonStreamFile $file = null;
2627

@@ -30,7 +31,7 @@ public function execute(ProcessState $state): void
3031
$options = $this->getOptions($state);
3132
$this->file = new JsonStreamFile(
3233
$options['file_path'],
33-
'wb',
34+
$options['mode'],
3435
$options['spl_file_object_flags'],
3536
$options['json_flags'],
3637
);
@@ -49,6 +50,14 @@ public function proceed(ProcessState $state): void
4950
$state->setOutput($options['file_path']);
5051
}
5152

53+
/**
54+
* Release the file at the end of the process, so that the task can write it again on its next execution.
55+
*/
56+
public function finalize(ProcessState $state): void
57+
{
58+
$this->file = null;
59+
}
60+
5261
protected function configureOptions(OptionsResolver $resolver): void
5362
{
5463
$resolver->setRequired(['file_path']);
@@ -66,9 +75,11 @@ protected function configureOptions(OptionsResolver $resolver): void
6675
)
6776
);
6877
$resolver->setDefaults([
78+
'mode' => 'wb',
6979
'spl_file_object_flags' => null,
7080
'json_flags' => null,
7181
]);
82+
$resolver->setAllowedTypes('mode', ['string']);
7283
$resolver->setAllowedTypes('spl_file_object_flags', ['array', 'null']);
7384
$resolver->setAllowedTypes('json_flags', ['array', 'null']);
7485
}
Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
/*
6+
* This file is part of the CleverAge/ProcessBundle package.
7+
*
8+
* Copyright (c) Clever-Age
9+
*
10+
* For the full copyright and license information, please view the LICENSE
11+
* file that was distributed with this source code.
12+
*/
13+
14+
namespace CleverAge\ProcessBundle\Task\File\Xml;
15+
16+
use CleverAge\ProcessBundle\Model\AbstractConfigurableTask;
17+
use CleverAge\ProcessBundle\Model\FinalizableTaskInterface;
18+
use CleverAge\ProcessBundle\Model\IterableTaskInterface;
19+
use CleverAge\ProcessBundle\Model\ProcessState;
20+
use Symfony\Component\OptionsResolver\OptionsResolver;
21+
22+
/**
23+
* Read an XML file element by element with \XMLReader, so that the whole file is never loaded in memory.
24+
*/
25+
class XmlStreamReaderTask extends AbstractConfigurableTask implements IterableTaskInterface, FinalizableTaskInterface
26+
{
27+
protected ?\XMLReader $reader = null;
28+
29+
protected ?string $filePath = null;
30+
31+
public function execute(ProcessState $state): void
32+
{
33+
$nodeName = (string) $this->getOption($state, 'node_name');
34+
if ($this->reader instanceof \XMLReader) {
35+
// Skip the subtree of the current element: nested elements with the same name are not iterated
36+
$found = $this->moveToElement($nodeName, true);
37+
} else {
38+
$this->open($this->getFilePath($state));
39+
$found = $this->moveToElement($nodeName, false);
40+
}
41+
42+
if (!$found) {
43+
// Release the file to allow the following iteration to read it again
44+
$this->close();
45+
$state->setSkipped(true);
46+
47+
return;
48+
}
49+
50+
$state->setOutput($this->expand());
51+
}
52+
53+
/**
54+
* The reader is moved in execute(), so that XML errors are handled by the error strategy of the task.
55+
*/
56+
public function next(ProcessState $state): bool
57+
{
58+
return $this->reader instanceof \XMLReader;
59+
}
60+
61+
public function finalize(ProcessState $state): void
62+
{
63+
$this->close();
64+
}
65+
66+
protected function getFilePath(ProcessState $state): string
67+
{
68+
$filePath = $this->getOption($state, 'file_path') ?? $state->getInput();
69+
if (!\is_string($filePath) || '' === $filePath) {
70+
throw new \UnexpectedValueException('No file path given in the "file_path" option or as input');
71+
}
72+
73+
return $filePath;
74+
}
75+
76+
protected function open(string $filePath): void
77+
{
78+
if (!is_file($filePath) || !is_readable($filePath)) {
79+
throw new \UnexpectedValueException(\sprintf('XML file "%s" does not exist or is not readable', $filePath));
80+
}
81+
82+
$reader = new \XMLReader();
83+
if (!@$reader->open($filePath)) {
84+
throw new \RuntimeException(\sprintf('Could not open XML file "%s"', $filePath));
85+
}
86+
$this->reader = $reader;
87+
$this->filePath = $filePath;
88+
}
89+
90+
/**
91+
* Move the reader to the next element named $nodeName, return false at the end of the file.
92+
*/
93+
protected function moveToElement(string $nodeName, bool $skipCurrent): bool
94+
{
95+
$reader = $this->getReader();
96+
97+
$previousUseErrors = libxml_use_internal_errors(true);
98+
libxml_clear_errors();
99+
try {
100+
$found = $skipCurrent ? $reader->next() : $reader->read();
101+
while ($found && (\XMLReader::ELEMENT !== $reader->nodeType || $reader->name !== $nodeName)) {
102+
$found = $reader->read();
103+
}
104+
$errors = libxml_get_errors();
105+
} finally {
106+
libxml_clear_errors();
107+
libxml_use_internal_errors($previousUseErrors);
108+
}
109+
110+
$this->throwOnErrors($errors);
111+
112+
return $found;
113+
}
114+
115+
protected function expand(): \DOMElement
116+
{
117+
$reader = $this->getReader();
118+
$document = new \DOMDocument();
119+
120+
$previousUseErrors = libxml_use_internal_errors(true);
121+
libxml_clear_errors();
122+
try {
123+
$node = $reader->expand($document);
124+
$errors = libxml_get_errors();
125+
} finally {
126+
libxml_clear_errors();
127+
libxml_use_internal_errors($previousUseErrors);
128+
}
129+
130+
$this->throwOnErrors($errors);
131+
if (!$node instanceof \DOMElement) {
132+
throw new \UnexpectedValueException(\sprintf('Could not expand element "%s" of XML file "%s"', $reader->name, $this->filePath));
133+
}
134+
135+
// Each element gets its own document, so that it can be evaluated with xpath like the output of XmlReaderTask
136+
$document->appendChild($node);
137+
138+
return $node;
139+
}
140+
141+
/**
142+
* Warnings are tolerated, errors (e.g. mismatched tags) and fatal errors are not.
143+
*
144+
* @param array<\LibXMLError> $errors
145+
*/
146+
protected function throwOnErrors(array $errors): void
147+
{
148+
$errors = array_filter($errors, static fn (\LibXMLError $error): bool => \LIBXML_ERR_WARNING !== $error->level);
149+
if ([] === $errors) {
150+
return;
151+
}
152+
153+
$messages = array_map(
154+
static fn (\LibXMLError $error): string => \sprintf('%s (line %d, column %d)', trim($error->message), $error->line, $error->column),
155+
$errors,
156+
);
157+
$filePath = $this->filePath;
158+
$this->close();
159+
160+
throw new \UnexpectedValueException(\sprintf('Invalid XML in file "%s": %s', $filePath, implode('; ', $messages)));
161+
}
162+
163+
protected function getReader(): \XMLReader
164+
{
165+
if (!$this->reader instanceof \XMLReader) {
166+
throw new \LogicException('No XML file opened');
167+
}
168+
169+
return $this->reader;
170+
}
171+
172+
protected function close(): void
173+
{
174+
$this->reader?->close();
175+
$this->reader = null;
176+
$this->filePath = null;
177+
}
178+
179+
protected function configureOptions(OptionsResolver $resolver): void
180+
{
181+
$resolver->setDefault('file_path', null);
182+
$resolver->setAllowedTypes('file_path', ['string', 'null']);
183+
184+
$resolver->setRequired('node_name');
185+
$resolver->setAllowedTypes('node_name', 'string');
186+
}
187+
}

0 commit comments

Comments
 (0)