Sumo User Mailing List

[sumo-user] NETCONVERT freezes on very large networks

Classic

List

Threaded

12 messages Options

Sasan Amini-2

[sumo-user] NETCONVERT freezes on very large networks

Dear all,

I am trying to convert a very large OSM file to a SUMO network using NETCONVERT. When I set --geometry.remove, the processes freezes when trying to remove empty nodes and never ends. Is this a memory issue or somehow NETCONVERT doesn't like to remove that many nodes? btw, I have 40GB of RAM and 12 CPU cores and without --geometry.remove I get the network in less than 5 minutes.

Thanks,

Sasan

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Jakob Erdmann

Re: [sumo-user] NETCONVERT freezes on very large networks

My guess would be that this isn't about size but somewhere in that big network is a piece of input which triggers a bug. You could try to split your input in half and convert each individually to check this hypothesis.

regards,

Jakob

Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini <[hidden email]>:

Dear all,

I am trying to convert a very large OSM file to a SUMO network using NETCONVERT. When I set --geometry.remove, the processes freezes when trying to remove empty nodes and never ends. Is this a memory issue or somehow NETCONVERT doesn't like to remove that many nodes? btw, I have 40GB of RAM and 12 CPU cores and without --geometry.remove I get the network in less than 5 minutes.

Thanks,
Sasan

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

behrisch

Re: [sumo-user] NETCONVERT freezes on very large networks

Administrator

Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:

> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Sasan Amini-2

Re: [sumo-user] NETCONVERT freezes on very large networks

True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:

Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Jakob Erdmann

Re: [sumo-user] NETCONVERT freezes on very large networks

How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:

True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Sasan Amini-2

Re: [sumo-user] NETCONVERT freezes on very large networks

I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:

How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Jakob Erdmann

Re: [sumo-user] NETCONVERT freezes on very large networks

I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:

I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Sasan Amini-2

Re: [sumo-user] NETCONVERT freezes on very large networks

I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:

I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Jakob Erdmann

Re: [sumo-user] NETCONVERT freezes on very large networks

I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.

can you please report

- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)

- the number of junctions in either network (verbose output lists them in the node type statistics)

- the final network size when removing/not removing nodes.

regards,

Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:

I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Sasan Amini-2

Re: [sumo-user] NETCONVERT freezes on very large networks

The size of the network before processing:

831430 nodes and 1441826 edges. I didn't find the final network size in the log file and didn't save each network separately, so can't be sure if I can say the size of the resulted network after processing :-/

Here are the other two statistics (in the sequence of the above notebook cells)

1- 0 Nodes removed.

Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9354
Priority junctions : 802752
Right-before-left junctions : 1830
Traffic light junctions : 16969

2- 327343 nodes removed.

Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9352
Priority junctions : 381203
Right-before-left junctions : 1746
Traffic light junctions : 111261

3- 425644 nodes removed.

Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9340
Priority junctions : 381589
Right-before-left junctions : 1822
Traffic light junctions : 12510

4- 425644 nodes removed.

Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9340
Priority junctions : 377231
Right-before-left junctions : 1799
Traffic light junctions : 16891

Best,

Sasan

On Mon, Mar 1, 2021 at 12:21 PM Jakob Erdmann <[hidden email]> wrote:

I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Jakob Erdmann

Re: [sumo-user] NETCONVERT freezes on very large networks

Can you make your input file available? I stilll think there is a bug somewhere.

Am Di., 2. März 2021 um 11:09 Uhr schrieb Sasan Amini <[hidden email]>:

The size of the network before processing:
831430 nodes and 1441826 edges. I didn't find the final network size in the log file and didn't save each network separately, so can't be sure if I can say the size of the resulted network after processing :-/
Here are the other two statistics (in the sequence of the above notebook cells)
1- 0 Nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9354
Priority junctions : 802752
Right-before-left junctions : 1830
Traffic light junctions : 16969
2-  327343 nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9352
Priority junctions : 381203
Right-before-left junctions : 1746
Traffic light junctions : 111261
3-  425644 nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9340
Priority junctions : 381589
Right-before-left junctions : 1822
Traffic light junctions : 12510
4-   425644 nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9340
Priority junctions : 377231
Right-before-left junctions : 1799
Traffic light junctions : 16891

Best,
Sasan



On Mon, Mar 1, 2021 at 12:21 PM Jakob Erdmann <[hidden email]> wrote:
I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

Jakob Erdmann

Re: [sumo-user] NETCONVERT freezes on very large networks

Thanks for the example. As it turns out, the slow-down was due to inefficient treatment of traffic lights after all (while your options did remove some traffic lights they also added new ones).

The issue is fixed in the latest development version.

regards,

Jakob

Am Di., 2. März 2021 um 11:55 Uhr schrieb Jakob Erdmann <[hidden email]>:

Can you make your input file available? I stilll think there is a bug somewhere.

Am Di., 2. März 2021 um 11:09 Uhr schrieb Sasan Amini <[hidden email]>:
The size of the network before processing:
831430 nodes and 1441826 edges. I didn't find the final network size in the log file and didn't save each network separately, so can't be sure if I can say the size of the resulted network after processing :-/
Here are the other two statistics (in the sequence of the above notebook cells)
1- 0 Nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9354
Priority junctions : 802752
Right-before-left junctions : 1830
Traffic light junctions : 16969
2-  327343 nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9352
Priority junctions : 381203
Right-before-left junctions : 1746
Traffic light junctions : 111261
3-  425644 nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9340
Priority junctions : 381589
Right-before-left junctions : 1822
Traffic light junctions : 12510
4-   425644 nodes removed.
Summary:
Node type statistics:
Unregulated junctions : 0
Dead-end junctions : 9340
Priority junctions : 377231
Right-before-left junctions : 1799
Traffic light junctions : 16891

Best,
Sasan



On Mon, Mar 1, 2021 at 12:21 PM Jakob Erdmann <[hidden email]> wrote:
I find these values quite surprising since the running time for node removal should be linear in the removed nodes when there are no traffic lights and quite low compared to all the other computations that are going on.
can you please report
- the number of nodes removed (verbose output should say: Removing empty nodes and geometry nodes. X nodes removed)
- the number of junctions in either network (verbose output lists them in the node type statistics)
- the final network size when removing/not removing nodes.

regards,
Jakob

Am Mo., 1. März 2021 um 11:51 Uhr schrieb Sasan Amini <[hidden email]>:
I tried some variations, you can see the times below. It got generally faster after I rebooted my computer. --geometry.remove is by far the most influential factor:

On Wed, Feb 24, 2021 at 5:16 PM Jakob Erdmann <[hidden email]> wrote:
I still suspect a bug rather than a "simple" algorithm-to-slow problem. To rule out one piece of slow-algorithm in the context of geometry.remove, can you try running with option --tls.discard-loaded?

Am Mi., 24. Feb. 2021 um 16:28 Uhr schrieb Sasan Amini <[hidden email]>:
I didn't measure the time in my script, but it wasn't more than 5 minutes. and the network is entire Germany's road network from OSM down to "highway.secondary".

On Wed, Feb 24, 2021 at 4:26 PM Jakob Erdmann <[hidden email]> wrote:
How long does it run without --geometry.remove?

Am Mi., 24. Feb. 2021 um 16:24 Uhr schrieb Sasan Amini <[hidden email]>:
True. It ended after >36 hours. I am using version 1.7.0 actually.

On Tue, Feb 23, 2021 at 10:36 PM Michael Behrisch <[hidden email]> wrote:
Hi,
if it occurs only with the latest nightly build but not with 1.8 it
might as well be https://github.com/eclipse/sumo/issues/8270. It does
not freeze it just takes very long, I also noticed that with the Berlin
network.

Best regards,
Michael

Am 19.02.21 um 07:48 schrieb Jakob Erdmann:
> My guess would be that this isn't about size but somewhere in that big
> network is a piece of input which triggers a bug. You could try to split
> your input in half and convert each individually to check this hypothesis.
>
> regards,
> Jakob
>
>
> Am Do., 18. Feb. 2021 um 21:34 Uhr schrieb Sasan Amini
> <[hidden email] <mailto:[hidden email]>>:
>
> Dear all,
>
> I am trying to convert a very large OSM file to a SUMO network using
> NETCONVERT. When I set --geometry.remove, the processes freezes when
> trying to remove empty nodes and never ends. Is this a memory issue
> or somehow NETCONVERT doesn't like to remove that many nodes? btw, I
> have 40GB of RAM and 12 CPU cores and without --geometry.remove I
> get the network in less than 5 minutes.
>
> Thanks,
> Sasan
> image.png
> _______________________________________________
> sumo-user mailing list
> [hidden email] <mailto:[hidden email]>
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/sumo-user
> <https://www.eclipse.org/mailman/listinfo/sumo-user>
>
>
> _______________________________________________
> sumo-user mailing list
> [hidden email]
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user
>

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user

_______________________________________________
sumo-user mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/sumo-user