[sumo-user] NETCONVERT freezes on very large networks

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[sumo-user] NETCONVERT freezes on very large networks

Sasan Amini-2
Dear all,

I am trying to convert a very large OSM file to a SUMO network using NETCONVERT. When I set --geometry.remove, the processes freezes when trying to remove empty nodes and never ends. Is this a memory issue or somehow NETCONVERT doesn't like to remove that many nodes? btw, I have 40GB of RAM and 12 CPU cores and without --geometry.remove I get the network in less than 5 minutes.

Thanks,
Sasan
image.png

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Jakob Erdmann
My guess would be that this isn't about size but somewhere in that big network is a piece of input which triggers a bug. You could try to split your input in half and convert each individually to check this hypothesis.

regards,
Jakob


Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini <[hidden email]>:
Dear all,

I am trying to convert a very large OSM file to a SUMO network using NETCONVERT. When I set --geometry.remove, the processes freezes when trying to remove empty nodes and never ends. Is this a memory issue or somehow NETCONVERT doesn't like to remove that many nodes? btw, I have 40GB of RAM and 12 CPU cores and without --geometry.remove I get the network in less than 5 minutes.

Thanks,
Sasan
image.png
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

behrisch
Administrator
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:

> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Sasan Amini-2
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Jakob Erdmann
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Sasan Amini-2
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Jakob Erdmann
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Sasan Amini-2
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:
image.png

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Jakob Erdmann
I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:
image.png

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Sasan Amini-2
The size of the network before processing:
831430 nodes and 1441826 edges. I didn't find the final network size in the log file and didn't save each network separately, so can't be sure if I can say the size of the resulted network after processing :-/
Here are the other two statistics (in the sequence of the above notebook cells)
1- 0 Nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9354
  Priority junctions          : 802752
  Right-before-left junctions : 1830
  Traffic light junctions      : 16969
2-  327343 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9352
  Priority junctions          : 381203
  Right-before-left junctions : 1746
  Traffic light junctions      : 111261
3-  425644 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9340
  Priority junctions          : 381589
  Right-before-left junctions : 1822
  Traffic light junctions      : 12510
4-    425644 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9340
  Priority junctions          : 377231
  Right-before-left junctions : 1799
  Traffic light junctions      : 16891

Best,
Sasan

    

On Mon, Mar 1, 2021 at 12:21 PM Jakob Erdmann <[hidden email]> wrote:
I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:
image.png

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Jakob Erdmann
Can you make your input file available? I stilll think there is a bug somewhere.

Am Di., 2. März 2021 um 11:09 Uhr schrieb Sasan Amini <[hidden email]>:
The size of the network before processing:
831430 nodes and 1441826 edges. I didn't find the final network size in the log file and didn't save each network separately, so can't be sure if I can say the size of the resulted network after processing :-/
Here are the other two statistics (in the sequence of the above notebook cells)
1- 0 Nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9354
  Priority junctions          : 802752
  Right-before-left junctions : 1830
  Traffic light junctions      : 16969
2-  327343 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9352
  Priority junctions          : 381203
  Right-before-left junctions : 1746
  Traffic light junctions      : 111261
3-  425644 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9340
  Priority junctions          : 381589
  Right-before-left junctions : 1822
  Traffic light junctions      : 12510
4-    425644 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9340
  Priority junctions          : 377231
  Right-before-left junctions : 1799
  Traffic light junctions      : 16891

Best,
Sasan

    

On Mon, Mar 1, 2021 at 12:21 PM Jakob Erdmann <[hidden email]> wrote:
I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:
image.png

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
Reply | Threaded
Open this post in threaded view
|

Re: [sumo-user] NETCONVERT freezes on very large networks

Jakob Erdmann
Thanks for the example. As it turns out, the slow-down was due to inefficient treatment of traffic lights after all (while your options did remove some traffic lights they also added new ones).
The issue is fixed in the latest development version.
regards,
Jakob

Am Di., 2. März 2021 um 11:55 Uhr schrieb Jakob Erdmann <[hidden email]>:
Can you make your input file available? I stilll think there is a bug somewhere.

Am Di., 2. März 2021 um 11:09 Uhr schrieb Sasan Amini <[hidden email]>:
The size of the network before processing:
831430 nodes and 1441826 edges. I didn't find the final network size in the log file and didn't save each network separately, so can't be sure if I can say the size of the resulted network after processing :-/
Here are the other two statistics (in the sequence of the above notebook cells)
1- 0 Nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9354
  Priority junctions          : 802752
  Right-before-left junctions : 1830
  Traffic light junctions      : 16969
2-  327343 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9352
  Priority junctions          : 381203
  Right-before-left junctions : 1746
  Traffic light junctions      : 111261
3-  425644 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9340
  Priority junctions          : 381589
  Right-before-left junctions : 1822
  Traffic light junctions      : 12510
4-    425644 nodes removed.
Summary:
 Node type statistics:
  Unregulated junctions       : 0
  Dead-end junctions          : 9340
  Priority junctions          : 377231
  Right-before-left junctions : 1799
  Traffic light junctions      : 16891

Best,
Sasan

    

On Mon, Mar 1, 2021 at 12:21 PM Jakob Erdmann <[hidden email]> wrote:
I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:
image.png

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually. 

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
>     Dear all,
>
>     I am trying to convert a very large OSM file to a SUMO network using
>     NETCONVERT. When I set --geometry.remove, the processes freezes when
>     trying to remove empty nodes and never ends. Is this a memory issue
>     or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
>     have 40GB of RAM and 12 CPU cores and without --geometry.remove I
>     get the network in less than 5 minutes.
>
>     Thanks,
>     Sasan
>     image.png
>     _______________________________________________
>     sumo-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/sumo-user
>     <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user