A lot of data is missing for Germany
Good evening!
For Germany, many and significantly important data have been missing for several weeks! This is the case as of the data slice of 07.08.2023.
Thank you.
Sven
Pinned replies
Replies 24
fyi: We now experience a bigger problem: France. France was always complicated but now it has a land mass as well breaking the tree more severe than all the other examples I looked at in the past.
This will result to the fact that France Metropolitaine (3) is within France (2) but all Regions (4)
are within France (Landmass) (2), but should be normally within
France Metropolitaine (3). This issue is even more severe and complicated to fix as it was with other countries because they were mostly only affecting the first child-level after the country (2) itself.
In the long run I don't see any other way than to exclude `boundary=land_area` from the country tree or only include `boundary=administrative` in the tree.
According to my tests, in the most recent dataset (`osm20240205`), the following Features are affected (at the tree top level):
|correct_osm_id|correct_name |land_area_osm_id|land_area_name |
|--------------|---------------------------|----------------|---------------------------------------|
|-536,900 |Antigua and Barbuda |-9,604,462 |Antigua and Barbuda (land mass) |
|-52,411 |België / Belgique / Belgien|-937,244 |België / Belgique / Belgien (land mass)|
|-79,510 |Eesti |-4,463,372 |Eesti (land area) |
|-2,202,162 |France |-16,467,322 |France (terres) |
|-270,009 |Guernsey |-6,571,872 |Guernsey (land mass) |
|-367,988 |Jersey |-1,711,283 |Jersey (land mass) |
|-72,596 |Lietuva |-4,474,651 |Lietuva (sausuma) |
|-1,124,039 |Monaco |-36,990 |Monaco |
|-49,715 |Polska |-936,128 |Polska (ląd) |
|-305,095 |قطر |-3,832,630 |قطر (الكتلة الأرضية) |
|-87,565 |South Africa |-1,252,792 |South Africa (land mass) |
The costline was broken in the planet.osm from that date. You can see it in the last column on the Status page. 30% of the world's land mass is gone.
Don't use that database if you care about the land mass. You can most likely use osm20240304 instead, which is newer.
fyi: We now experience a bigger problem: France. France was always complicated but now it has a land mass as well breaking the tree more severe than all the other examples I looked at in the past.
This will result to the fact that France Metropolitaine (3) is within France (2) but all Regions (4)
are within France (Landmass) (2), but should be normally within
France Metropolitaine (3). This issue is even more severe and complicated to fix as it was with other countries because they were mostly only affecting the first child-level after the country (2) itself.
In the long run I don't see any other way than to exclude `boundary=land_area` from the country tree or only include `boundary=administrative` in the tree.
According to my tests, in the most recent dataset (`osm20240205`), the following Features are affected (at the tree top level):
|correct_osm_id|correct_name |land_area_osm_id|land_area_name |
|--------------|---------------------------|----------------|---------------------------------------|
|-536,900 |Antigua and Barbuda |-9,604,462 |Antigua and Barbuda (land mass) |
|-52,411 |België / Belgique / Belgien|-937,244 |België / Belgique / Belgien (land mass)|
|-79,510 |Eesti |-4,463,372 |Eesti (land area) |
|-2,202,162 |France |-16,467,322 |France (terres) |
|-270,009 |Guernsey |-6,571,872 |Guernsey (land mass) |
|-367,988 |Jersey |-1,711,283 |Jersey (land mass) |
|-72,596 |Lietuva |-4,474,651 |Lietuva (sausuma) |
|-1,124,039 |Monaco |-36,990 |Monaco |
|-49,715 |Polska |-936,128 |Polska (ląd) |
|-305,095 |قطر |-3,832,630 |قطر (الكتلة الأرضية) |
|-87,565 |South Africa |-1,252,792 |South Africa (land mass) |
Thanks for all the work you are doing to maintain this service!
We ran into the same issue, especially with the boundaries in Germany and Poland. In the further processing, we use the parents part of the data quite heavily so unfortunately, we can't just download more data and our problem would be solved. Currently the download selection for Germany, admin level 2 to 8 gives us this URL: https://osm-boundaries.com/Download/Submit?apiKey=…&db=osm20230904&osmIds=-51477&recursive&minAdminLevel=2&maxAdminLevel=8&format=GeoJSON&srid=4326. This still gives us recursively only all boundaries that are (in the tree) within Germany (not land mass relation). Which means we only get 562 features instead of the 12532 features we got in the April database.
For us it would be very helpful if there would be a version that only considers boundary=administrative features in OSM, as already suggested in the thread earlier. If understand it correctly it would solve all our issues as we only use these boundaries in the later processing anyway.
Is there any way we could contribute to achieve this goal? As far as I know the code is not open source (yet).
Regards
Johannes
Screenshot how the aforementioned API url was created:
No the code is closed source, and for now it will be kept like that. Therefore there aren't any real ways to contribute vs this.
It may sound like a small task, but it's a bit of a fundamental change to provide alternative tree views. One could wish that we had the idea from the beginning, then it would be easier now. It is planned though, but haven't had any time-frame at all. Now with this issue there is at least reason to prioritize it more. But I still wouldn't expect it to happen this year. As mentioned earlier, this site is a loss from an economical perspective and it's therefore very hard for us to prioritize it before other tasks that we have.
Thanks for all the work you are doing to maintain this service!
We ran into the same issue, especially with the boundaries in Germany and Poland. In the further processing, we use the parents part of the data quite heavily so unfortunately, we can't just download more data and our problem would be solved. Currently the download selection for Germany, admin level 2 to 8 gives us this URL: https://osm-boundaries.com/Download/Submit?apiKey=…&db=osm20230904&osmIds=-51477&recursive&minAdminLevel=2&maxAdminLevel=8&format=GeoJSON&srid=4326. This still gives us recursively only all boundaries that are (in the tree) within Germany (not land mass relation). Which means we only get 562 features instead of the 12532 features we got in the April database.
For us it would be very helpful if there would be a version that only considers boundary=administrative features in OSM, as already suggested in the thread earlier. If understand it correctly it would solve all our issues as we only use these boundaries in the later processing anyway.
Is there any way we could contribute to achieve this goal? As far as I know the code is not open source (yet).
Regards
Johannes
Screenshot how the aforementioned API url was created:
Offer to share information and correlate data.
...I would still like to make you this offer... But please do it in the forum: https://community.openstreetmap.org
This makes it easier for me to write directly in German and, above all, it involves a much larger circle of interested parties! (this is extremely important here!)
I have also looked around in neighbouring countries... It's similar there. In Poland, for example, there is a very strong allocation to churches, recognisable in your data by deanery=Dekanat* Example: https://www.openstreetmap.org/relation/15910987
In your data, this is assigned in a wide variety of places!
...sometimes directly, sometimes at different levels of the administrative structure!
Another example is that you assign protected areas to administrative borders in a completely arbitrary way. I don't have to look at Germany, that's just the way it is in neighbouring countries.
I only want data that can be used and structured to the best of our knowledge. If it takes a while, I don't care.Sven
my concluding words:
You want to present as much and everything as possible on borders. Good. But you are losing sight of the complexity of all the borders!
...I have already written that land mass has nothing to do with administrative borders, land mass is not a boundary=administrative!!!!
There is a lack of structure...
For example, there is a lack of primary distinction from purely administrative borders (boundary=administrarive) to protected area borders...
For example, in your data relation https://www.openstreetmap.org/relation/4763316 is assigned to relation https://www.openstreetmap.org/relation/1388880. This may be correct in terms of location, but in terms of content it is total nonsense. Relation 4763316 is protect_class=4 and at most assigned to admin_level=4 (Brandenburg), but this is also only partly true. It is actually a completely separate area... That's how it runs through all the data!
Please concentrate primarily on the administrative limits, make sure that the data imports always work properly and make sure that the import intervals are shorter: for example, at least once a week!
This will help OSM a lot more!
The original idea was to use boundary=administrative. But since the site we develop, that uses data from OSM-Boundaries, uses data for all countries in the world, not just Germany, we also were hit by reality. The quality of the data in OSM isn't perfect. We needed data that wasn't properly tagged, and therefore had to add other polygon data as well. Again, the site is built upon our need primarily. We are just providing what we built, for free.
If we only show data that has boundary=administrative the site might be useful for you, but it would then by useless to us. We need a lot of polygon data that isn't tagged like that. For various reasons, but most often because less developed countries than Germany doesn't have as reliable data.
But it sounds like what you want is what I have suggested several times. A different tree, which only shows boundary=administrative.
And again, last time, the data imports work properly. The imports work exactly as planned and designed. You just don't seem to understand the rules that we set up, and why they are as they are. The imports works. And the data is as we wished it to be. We just wish that the OSM data was less of a Wild West.
And about an import every week. Sorry, that most likely won't happen. I don't think you appreciate the amount of data we process here. We spend 3-8 weeks of CPU power for every import we do. If you have used the site a lot you should have noticed, and read about, the processing time we need. This also tells us that it's theoretically impossible for us to do imports more often than we do. In fact, it might be that we have to do it less often, not the least when considering the increase in size of planet.osm by time. This can of course change, if someone chooses to fund us with the attached costs. If someone would be interested in that we would have to look into that, but it would most likely be north of 1000 EUR/month.
Oh...
...still forgotten...
If you have further questions, don't be afraid to ask them at https://community.openstreetmap.org/ ! You know that we here in Germany have a lot of knowledge about this...corresponding answers are guaranteed at https://community.openstreetmap.org/c/communities/de/56
my concluding words:
You want to present as much and everything as possible on borders. Good. But you are losing sight of the complexity of all the borders!
...I have already written that land mass has nothing to do with administrative borders, land mass is not a boundary=administrative!!!!
There is a lack of structure...
For example, there is a lack of primary distinction from purely administrative borders (boundary=administrarive) to protected area borders...
For example, in your data relation https://www.openstreetmap.org/relation/4763316 is assigned to relation https://www.openstreetmap.org/relation/1388880. This may be correct in terms of location, but in terms of content it is total nonsense. Relation 4763316 is protect_class=4 and at most assigned to admin_level=4 (Brandenburg), but this is also only partly true. It is actually a completely separate area... That's how it runs through all the data!
Please concentrate primarily on the administrative limits, make sure that the data imports always work properly and make sure that the import intervals are shorter: for example, at least once a week!
This will help OSM a lot more!
Then use another database. We serve data as is in OSM. If the data is broken in OSM it's broken in our end as well. We do not spend any time to try to fix it, that would be financially impossible for us.
OSM always has broken boundary polygons somewhere, that's why there are several databases served.
See Multiple databases under https://osm-boundaries.com/Documentation .