The Amazing Race!
原文链接: https://www.nv5geospatialsoftware.com/Learn/Blogs/Blog-Details/the-amazing-race
13977 Rate this article:
.1
The Amazing Race!
Zachary Norman Wednesday, September 28, 2016
It wasn't so long ago that the IDL-Python bridge was introduced to IDL 8.5. It was with this new version, that I got my first experience programming with Python and testing the IDL-Python bridge. Through the past year it has been exciting to see the new changes and improvements that have become a part of the bridge.
Some of these new features include:
-Better error catching with the IDL-Python bridge
-Enhanced Jupyter notebook that allows for the development of full IDL programs and Python code in the same environment
-Improved support for variables passing back and forth
With all the time I have spent with Python, I have always wondered what some of the advantages are between Python and IDL. One thing that I have commonly heard several engineers say was that IDL was much faster than Python. For this blog, I decided to put that to the test and see how Python and IDL really compared to one another.
Before talking about the test, I do just want to explain things a bit about how it was set up and some potential caveats about the processing times that will be shown. With the tests I created, I did my best to choose tests that were comparable between IDL and Python. Since I'm no expert at Python, there very well may have been other methods that could be faster than what I will show. Most of the pieces I included in the test are things I found easily by doing a web search - meaning that most of the approaches I used were the most common programming methods that people are likely using. This shows how much faster IDL might be than a general program than something that someone might write in Python.
The test:
Here is what was actually tested between IDL and Python with an array of [10000,10000] or 10000*10000 elements
-Array creation time
-Type conversion times
-Index array creation times (i.e. [0,1,2,3,4...,n-1])
-Incrementing array values of all elements by 1
-Complex math expression with array (exact equation: sqrt(sin(arr*arr)))
-Single threaded for IDL and multithreaded
-Array element access times (i.e. setting y = arr[i])
-Simple image processing filter times (filters: sobel, roberts, prewitt)
The results:
Average array creation time (seconds):
Python : 0.213000 +/- 0.00953933
IDL : 0.0936666 +/- 0.0155028
Total time (seconds):
Python : 0.639000
IDL : 0.281000
Python/IDL time ratio: 2.27402
Average array data type conversion time (seconds):
Python : 0.171333 +/- 0.0155028
IDL : 0.0730000 +/- 0.00866031
Total time (seconds):
Python : 0.514000
IDL : 0.219000
Python/IDL time ratio: 2.34703
Average index array creation time (seconds):
Python : 0.229000 +/- 0.00866031
IDL : 0.124667 +/- 0.0160104
Total time (seconds):
Python : 0.687000
IDL : 0.374000
Python/IDL time ratio: 1.83690
Average increasing array value time (seconds):
Python : 0.0933333 +/- 0.000577446
IDL : 0.0313334 +/- 0.000577377
Total time (seconds):
Python : 0.280000
IDL : 0.0940001
Python/IDL time ratio: 2.97872
Average complex math statements (1 thread) time (seconds):
Python : 6.36967 +/- 0.0645319
IDL : 8.34667 +/- 0.0155028
Total time (seconds):
Python : 19.1090
IDL : 25.0400
Python/IDL time ratio: 0.763139
Average complex math statements (8 thread) time (seconds):
Python : 6.34400 +/- 0.0321871
IDL : 1.93933 +/- 0.00923762
Total time (seconds):
Python : 19.0320
IDL : 5.81800
Python/IDL time ratio: 3.27123
Average loop through array element time (seconds):
Python : 11.5290 +/- NaN
IDL : 3.29100 +/- NaN
Total time (seconds):
Python : 11.5290
IDL : 3.29100
Python/IDL time ratio: 3.50319
Average image processing routines time (seconds):
Python : 15.3660 +/- 0.0829635
IDL : 1.39900 +/- 0.0238955
Total time (seconds):
Python : 46.0980
IDL : 4.19700
Python/IDL time ratio: 10.9836
Conclusion:
In short, IDL significantly outperformed Python will all the speed tests apart from the complex math statement. However, IDL has access to built in multithreading for large arrays and, with multithreading enabled, IDL outperforms Python significantly when using all available cores.
Below is the IDL code used to compare the processing speed of IDL and Python. To use it you will need a few Python modules which can be found at the beginning of the procedure "python_test".
IDL Code:
pro printTimes, pythontimes, idltimes, TIMED = timed
compile_opt idl2
;check if we have a name for the process to print with the mean
if ~keyword_set(timed) then begin
add = ' time (seconds):'
endif else begin
add = timed + ' time (seconds):'
endelse
print, 'Average ' + add
print, ' Python : ' + strtrim(pythontimes.mean(),2) + ' +/- ' + strtrim(stddev(pythontimes),2)
print, ' IDL : ' + strtrim(idltimes.mean(),2) + ' +/- ' + strtrim(stddev(idltimes),2)
print, 'Total time (seconds):'
print, ' Python : ' + strtrim(total(pythontimes),2)
print, ' IDL : ' + strtrim(total(idltimes),2)
print, 'Python/IDL time ratio: ' + strtrim(total(pythontimes)/total(idltimes),2)
end
pro python_test
compile_opt idl2
;initialize Python
>>> 'import numpy as np'
>>> 'import time'
>>> 'import math'
>>> 'from idlpy import *'
>>> 'from skimage.filters import roberts, sobel, prewitt'
;number of times we want to run each test
nloops = 3
;array dimensions for iterator tests
dims = 10000
;>>>'dims = ' + strtrim(dims,2)
python.dims = dims
;array dimensions for filter tests
dims_filter = 10000
python.dims_filter = dims_filter
;initialize arrays to hol dpython times
pythontimes = fltarr(nloops)
idltimes = fltarr(nloops)
;test array creation in Python and IDL
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'arr = np.ones((dims,dims))'
tEndPython = systime(/seconds)
arr = fltarr(dims,dims)
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'array creation'
;check type conversion times
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'arr2 = arr.astype(int)'
tEndPython = systime(/seconds)
arr2 = fix(arr)
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'array data type conversion'
;check index array creation times
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'arr = np.arange(0, dims*dims, dtype=long)'
tEndPython = systime(/seconds)
arr = lindgen(dims*dims)
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'index array creation'
;check adding values to array
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'arr += 1'
tEndPython = systime(/seconds)
arr += 1
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'increasing array value'
;check complex math expressions with a single thread
pref_set, 'IDL_CPU_TPOOL_NTHREADS', 1, /commit
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'y = np.sin(arr*arr)**.5'
tEndPython = systime(/seconds)
y = sqrt(sin(arr*arr))
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'complex math statements (' + strtrim(!CPU.TPOOL_NTHREADS,2) + ' thread)'
pref_set, 'IDL_CPU_TPOOL_NTHREADS', /default, /commit
;check complex math expressions with all threads
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'y = np.sin(arr*arr)**.5'
tEndPython = systime(/seconds)
y = sqrt(sin(arr*arr))
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'complex math statements (' + strtrim(!CPU.TPOOL_NTHREADS,2) + ' thread)'
;check array element access times
nhere = 1
for i=0,nhere-1 do begin
tStart = systime(/seconds)
>>>'for x in np.nditer(arr):\n y=x'
tEndPython = systime(/seconds)
foreach x, arr do y = x
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes[0:nhere-1], idltimes[0:nhere-1], TIMED = 'loop through array element'
;check times for image processing
im_dat = lonarr(dims_filter, dims_filter)
;set a square in the middle of the image to 1
im_dat[.4*dims_filter:.6*dims_filter,.4*dims_filter:.6*dims_filter] = 1
;send the array to Python as well
python.im_dat = im_dat
for i=0,nloops-1 do begin
tStart = systime(/seconds)
>>> 'edge_sobel = sobel(im_dat)'
>>> 'edge_roberts = roberts(im_dat)'
>>> 'edge_prewitt = prewitt(im_dat)'
tEndPython = systime(/seconds)
edge_sobel = sobel(im_dat)
edge_roberts = roberts(im_dat)
edge_prewitt = prewitt(im_dat)
tEndIDL = systime(/seconds)
;save the times
pythontimes[i] = tEndPython - tStart
idltimes[i] = tEndIDL - tEndPython
endfor
print
printTimes, pythontimes, idltimes, TIMED = 'image processing routines'
stop
end
Coming Soon in ENVI 5.4: ENVITask Style Sheets Minimizing Rounding Errors with IDL's TOTAL function