osc_modif.py Questions and suggestions

[WK-GiHu]

Questions:

1. Will 'osc_modify.py' only called from 'update.py' or also invoked direct?

2. If also directly, which differs from the call by 'update.py',
   my first suggestion would be to harmonize all calls to 'osc_modify.osc_modify (...)'.

3. If only from update.py, I ignore the "if __name__ == __main__" block.

4. In 'try: ... except' block, the 'except' branch will not work if called from 'update.py',
   because "config.dbs" and "config.dpb" are not defined.

   The 'try' block may be omitted if 'options.osmbin_path' is always defined.
   But until now, I know the difference of "OsmBin" and "OsmOsis.OsmOsis" not yet.

[jocelynj]

1. The direct call for osc_modif.py is only for testing the diff generation. It is usually called directly by update.py. For information, cron contains the following lines:
   • ./update.py every minute, to generate diffs
   • ./update_pbf_all.sh every day, to update extracts from previously generated diffs
   • ./generate_merges.py every 10 minutes, to generate diffs for countries that are split on different continents
2. I don't really understand your comment. Maybe it is no longer relevant as this piece of code is only for testing ?

The difference between OsmBin and OsmOsis is:

• OsmBin uses a special database, that is storing only nodes locations, ways with their nodes, and full relation. These are stored on files, and is very fast to access when you only want to find coordinates for objects. It is updated every minute, at the end of update.py script.
• OsmOsis uses an osmosis database, which is generated by this tool. It is a direct "copy" of OSM information, containing all information on nodes, ways and relations, in a postgresql database.

[WK-GiHu]

I got 'osc_modify.unittest' running and added the Test Case #4 for the 'else' branch in osc_modify.osc_modify(...) .
Got the following results:

1. Python version <3.x, necessary!
2. In 'OsmSax.end()', line 607-614, is 'NodeGetPosition' called this function does not exist, however.
   Commented out so that I can continue testing.
3. In osc_modify.Test() line 136, 'assert' is always valid, because no file to 'tests/results'
   to be written. Have changed to 'os.stat(tests/out)'.
4. Error import in OsmGeom.py line 23-26 'from shapely. *'. Overloaded by 'dummy def's'.

TestCase Output:
`$python osc_modify.002
Creating node.crd
Creating way.idx
Creating way.data
Creating way.free
.
Begin TestCase # 1
new ways took: 1.09672546387e-05 for 0
End TestCase # 1
.
Begin TestCase # 2
read 60 nodes, 6 ways, 0 relations
End TestCase # 2
.
Begin TestCase # 3
[SIMULATE] OsmGeom.read_multipolygon(<open file 'tests/polygon.poly', mode 'r' at 0xf6de1e38>)
[SIMULATE] OsmGeom.check_intersection(poly.buffer,(18.3298116, 12.3082097))
read 82 nodes, 8 ways, 0 relations
End TestCase # 3
.
Begin TestCase # 4

End TestCase # 4

Ran 1 test in 0.134s
OK
`
Are the files at GitHub up to Date?
Missing 'NodeGetPosition' function to create itself, right?
Why is testing osc_modify.py.Test() the sub-directory 'tests/results'?
Missing modules 'from shapely. *', available?

TestCase # 4, I expected a empty file but got different content as in '001.osc'.
Still, I think this is an undefined option and should be reconsidered.

[jocelynj]

Can you push your new test somewhere so that I can take a look ?

I will check if the already present tests should work correctly or not.

[jocelynj]

I pushed three commits to fix the failures you had.

1. Yes, only python 2.7+ is supported. Did it cause an issue on your side ?
2. That's correct, this function is not defined in OsmBin.py, but it is defined on OsmOsis.py (it is the module used on self.reader). I think I've never found this bug because I've never used --position_only with an OsmBin database. It is easy to add support for this - will it make your test easier if I had this function to OsmBin ?
3. I've fixed it by creation directory if not existing. I've also updated the test files
4. Yes, a few dependencies are required. On my Debian, I've installed the following packages: python-nose python-lockfile python-shapely.

[WK-GiHu]

Still unanswered:
Why is testing osc_modify.py.Test() the sub-directory 'tests/results'?

TestCase # 4, I expected a empty file but got different content as in '001.osc'.
Still, I think this is an undefined option and should be reconsidered.

[WK-GiHu]

1. Python 2.7+, no issue, only to be aware not using 3.x syntax/code.
2. Using OsmOsis.NodeGetPosition(...) does not make sense.
   OsmOsis.NodeGetPosition are using SQL Statement.
   Compared with OsmBin.NodeGet(i) using FileIO.
   Assume a Typo, renamed self.reader.NodeGetPosition(i) into
   self.reader.NodeGet(i). Keep that in Mind if you ever will use --position_only
3. Makes no sense to compare static file signatur with a new created one.
   Added 'remove 'tests/out/*' at Test.setUp() and will stay for 'os.stat(tests/out)'.
4. Dependendcies resolved.
   Python-nose is a alternative for unittest.
   As update.py and osc_modify.py using unittest, maybe a future option.

Unittest is running without Error, Output:

`python osc_modif.002
remove tests/out/001.poly.osc
remove tests/out/001.dummy.osc
remove tests/out/001.bbox.osc
remove tests/out/001.pos.osc
Creating node.crd
Creating way.idx
Creating way.data
Creating way.free
.
Begin TestCase # 1
new ways took: 1.09672546387e-05 for 0
new nodes took: 0.0056459903717 for 36
End TestCase # 1
.
Begin TestCase # 2
read 60 nodes, 6 ways, 0 relations
End TestCase # 2
.
Begin TestCase # 3
read 51 nodes, 3 ways, 0 relations
End TestCase # 3
.
Begin TestCase # 4
End TestCase # 4
----------------------------------------------------------------------
Ran 1 test in 0.178s
OK
`

Try to upload osc_modif.002 to GitHub.