Discussion:
[Audacity-devel] Faster building (Linux with HDD)
James Crook
2017-04-09 14:08:27 UTC
Permalink
If you are building on linux with a hard drive rather than SSD, then you
may be able to get a worthwhile speed up in building using

make -j4

rather than just

make

This uses 4 cores for building rather than the default of 1. If you are
short of memory, you may want to close a browser if you have one open,
as browsers eat memory and more is used when using more cores to build.

I've seen discussions online that suggest you can use higher numbers
than the actual number of cores, and get further speed up IF building is
primarily disk bound. -j8 was suggested for a four core machine. It
was marginally less good for me on my 4 core machine than -j4.

I also read up on RAM disks, and for this use they apparently are a
waste of time.
Windows builds are multi-core by default.

I'm interested to hear what other people find.

--James.
Darrell Walisser
2017-04-09 15:57:58 UTC
Permalink
It's my experience that compilers are CPU bound, even with spinning rust.
The exception might be with extremely slow I/O (over network?) or
optimizations disabled.

So, I always use -j when compiling anything. In some cases memory can be
exhausted, but this is tolerable as I usually only swap for a few seconds
in one spot or two.

Note, there are times when the CPU can't max out because -j only forks
within a single Makefile. There are several spots in lib-src where this is
the case.

My test command (warning, git clean will nuke files you haven't added to
git!)
time sh -c 'git clean -dfx && ./configure && make -jX'
My results (6-core, 12-thread, SSD).

​A​t -j1, it takes.. OK I'm not even going there since it takes too long ;-)
At -j4, default optimization (-O2) Peak memory is about 1GB and it takes.
4:22 (12:42 CPU)
At -j12, peak is about 3GB and it takes 2:42 (16:23 CPU)
For -j12, -O3, peak is about 4GB and it takes 3:04 (20:30 CPU)

For a more realistic number I've included the ./configure part that doesn't
multi-thread. ./configure step seems like it would benefit from ram disk,
but:. SSD: 40.6s, ram disk: 40.2s. So, I can confirm they are a waste, at
least in the case of SSD.
If you are building on linux with a hard drive rather than SSD, then you
may be able to get a worthwhile speed up in building using
make -j4
rather than just
make
This uses 4 cores for building rather than the default of 1. If you are
short of memory, you may want to close a browser if you have one open,
as browsers eat memory and more is used when using more cores to build.
I've seen discussions online that suggest you can use higher numbers
than the actual number of cores, and get further speed up IF building is
primarily disk bound. -j8 was suggested for a four core machine. It
was marginally less good for me on my 4 core machine than -j4.
I also read up on RAM disks, and for this use they apparently are a
waste of time.
Windows builds are multi-core by default.
I'm interested to hear what other people find.
--James.
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
James Crook
2017-04-09 16:55:08 UTC
Permalink
Thanks Darrell.

I also tried ./configure -C
I saw many of the tests are being repeated. However that is getting
stuck in PORTAUDIO_CFLAGS not being cached. Since I mostly do make
without configure it isn't going to save me much. On Travis we only seem
to have one core, so make dominates the ./configure time, making it
marginal improving configure times there too.

--James.
Post by Darrell Walisser
It's my experience that compilers are CPU bound, even with spinning rust.
The exception might be with extremely slow I/O (over network?) or
optimizations disabled.
So, I always use -j when compiling anything. In some cases memory can be
exhausted, but this is tolerable as I usually only swap for a few seconds
in one spot or two.
Note, there are times when the CPU can't max out because -j only forks
within a single Makefile. There are several spots in lib-src where this is
the case.
My test command (warning, git clean will nuke files you haven't added to
git!)
time sh -c 'git clean -dfx && ./configure && make -jX'
My results (6-core, 12-thread, SSD).
​A​t -j1, it takes.. OK I'm not even going there since it takes too long ;-)
At -j4, default optimization (-O2) Peak memory is about 1GB and it takes.
4:22 (12:42 CPU)
At -j12, peak is about 3GB and it takes 2:42 (16:23 CPU)
For -j12, -O3, peak is about 4GB and it takes 3:04 (20:30 CPU)
For a more realistic number I've included the ./configure part that doesn't
multi-thread. ./configure step seems like it would benefit from ram disk,
but:. SSD: 40.6s, ram disk: 40.2s. So, I can confirm they are a waste, at
least in the case of SSD.
If you are building on linux with a hard drive rather than SSD, then you
may be able to get a worthwhile speed up in building using
make -j4
rather than just
make
This uses 4 cores for building rather than the default of 1. If you are
short of memory, you may want to close a browser if you have one open,
as browsers eat memory and more is used when using more cores to build.
I've seen discussions online that suggest you can use higher numbers
than the actual number of cores, and get further speed up IF building is
primarily disk bound. -j8 was suggested for a four core machine. It
was marginally less good for me on my 4 core machine than -j4.
I also read up on RAM disks, and for this use they apparently are a
waste of time.
Windows builds are multi-core by default.
I'm interested to hear what other people find.
--James.
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Darrell Walisser
2017-04-09 17:15:08 UTC
Permalink
Last I checked, Travis was supplying two cores. It also says so on their
doc page.
https://docs.travis-ci.com/user/ci-environment/

I got 14:49 when using two cores:
https://travis-ci.org/audacity/audacity/jobs/219332295. That's lower than
most of the builds, so it could be doing something. It's not easy to tell
since there is a lot of variation in build times.
Post by James Crook
Thanks Darrell.
I also tried ./configure -C
I saw many of the tests are being repeated. However that is getting stuck
in PORTAUDIO_CFLAGS not being cached. Since I mostly do make without
configure it isn't going to save me much. On Travis we only seem to have
one core, so make dominates the ./configure time, making it marginal
improving configure times there too.
--James.
It's my experience that compilers are CPU bound, even with spinning rust.
The exception might be with extremely slow I/O (over network?) or
optimizations disabled.
So, I always use -j when compiling anything. In some cases memory can be
exhausted, but this is tolerable as I usually only swap for a few seconds
in one spot or two.
Note, there are times when the CPU can't max out because -j only forks
within a single Makefile. There are several spots in lib-src where this is
the case.
My test command (warning, git clean will nuke files you haven't added to
git!)
time sh -c 'git clean -dfx && ./configure && make -jX'
My results (6-core, 12-thread, SSD).
​A​t -j1, it takes.. OK I'm not even going there since it takes too long ;-)
At -j4, default optimization (-O2) Peak memory is about 1GB and it takes.
4:22 (12:42 CPU)
At -j12, peak is about 3GB and it takes 2:42 (16:23 CPU)
For -j12, -O3, peak is about 4GB and it takes 3:04 (20:30 CPU)
For a more realistic number I've included the ./configure part that doesn't
multi-thread. ./configure step seems like it would benefit from ram disk,
but:. SSD: 40.6s, ram disk: 40.2s. So, I can confirm they are a waste, at
least in the case of SSD.
If you are building on linux with a hard drive rather than SSD, then you
may be able to get a worthwhile speed up in building using
make -j4
rather than just
make
This uses 4 cores for building rather than the default of 1. If you are
short of memory, you may want to close a browser if you have one open,
as browsers eat memory and more is used when using more cores to build.
I've seen discussions online that suggest you can use higher numbers
than the actual number of cores, and get further speed up IF building is
primarily disk bound. -j8 was suggested for a four core machine. It
was marginally less good for me on my 4 core machine than -j4.
I also read up on RAM disks, and for this use they apparently are a
waste of time.
Windows builds are multi-core by default.
I'm interested to hear what other people find.
--James.
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Steve the Fiddle
2017-04-09 20:05:57 UTC
Permalink
4 core i7-6700HQ CPU @ 2.60GHz
SSD
5GB of 8GB RAM free

make -j 8
183 seconds

Steve
Last I checked, Travis was supplying two cores. It also says so on their doc
page.
https://docs.travis-ci.com/user/ci-environment/
https://travis-ci.org/audacity/audacity/jobs/219332295. That's lower than
most of the builds, so it could be doing something. It's not easy to tell
since there is a lot of variation in build times.
Post by James Crook
Thanks Darrell.
I also tried ./configure -C
I saw many of the tests are being repeated. However that is getting stuck
in PORTAUDIO_CFLAGS not being cached. Since I mostly do make without
configure it isn't going to save me much. On Travis we only seem to have
one core, so make dominates the ./configure time, making it marginal
improving configure times there too.
--James.
It's my experience that compilers are CPU bound, even with spinning rust.
The exception might be with extremely slow I/O (over network?) or
optimizations disabled.
So, I always use -j when compiling anything. In some cases memory can be
exhausted, but this is tolerable as I usually only swap for a few seconds
in one spot or two.
Note, there are times when the CPU can't max out because -j only forks
within a single Makefile. There are several spots in lib-src where this is
the case.
My test command (warning, git clean will nuke files you haven't added to
git!)
time sh -c 'git clean -dfx && ./configure && make -jX'
My results (6-core, 12-thread, SSD).
At -j1, it takes.. OK I'm not even going there since it takes too long ;-)
At -j4, default optimization (-O2) Peak memory is about 1GB and it takes.
4:22 (12:42 CPU)
At -j12, peak is about 3GB and it takes 2:42 (16:23 CPU)
For -j12, -O3, peak is about 4GB and it takes 3:04 (20:30 CPU)
For a more realistic number I've included the ./configure part that doesn't
multi-thread. ./configure step seems like it would benefit from ram disk,
but:. SSD: 40.6s, ram disk: 40.2s. So, I can confirm they are a waste, at
least in the case of SSD.
If you are building on linux with a hard drive rather than SSD, then you
may be able to get a worthwhile speed up in building using
make -j4
rather than just
make
This uses 4 cores for building rather than the default of 1. If you are
short of memory, you may want to close a browser if you have one open,
as browsers eat memory and more is used when using more cores to build.
I've seen discussions online that suggest you can use higher numbers
than the actual number of cores, and get further speed up IF building is
primarily disk bound. -j8 was suggested for a four core machine. It
was marginally less good for me on my 4 core machine than -j4.
I also read up on RAM disks, and for this use they apparently are a
waste of time.
Windows builds are multi-core by default.
I'm interested to hear what other people find.
--James.
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
audacity-devel mailing list
https://lists.sourceforge.net/lists/listinfo/audacity-devel
Loading...