Vero - Wikipedia - User contributions [en]

Shellsort

2025-11-04T01:26:41Z

2605:8D80:5825:1532:D58:188C:5434:D54: /* Example */

{{Short description|Sorting algorithm which uses multiple comparison intervals}}
{{Infobox Algorithm
|class=[[Sorting algorithm]]
|image=[[File:Sorting shellsort anim.gif|Step-by-step visualisation of Shellsort]]
|caption=Shellsort with gaps 23, 10, 4, 1 in action
|data=[[Array data structure|Array]]
|time=O(''n''2) (worst known worst case gap sequence) O(''n'' log2''n'') (best known worst case gap sequence)<ref name="Pratt">{{Cite book
|last=Pratt
|first=Vaughan Ronald |author-link=Vaughan Ronald Pratt
|year=1979
|publisher=Garland
|title=Shellsort and Sorting Networks (Outstanding Dissertations in the Computer Sciences)
|url=https://apps.dtic.mil/sti/pdfs/AD0740110.pdf
|archive-url=https://web.archive.org/web/20210907132436/https://apps.dtic.mil/sti/pdfs/AD0740110.pdf
|url-status=live
|archive-date=7 September 2021
|isbn=978-0-8240-4406-0}}</ref>
|best-time=O(''n'' log ''n'') (most gap sequences) O(''n'' log2''n'') (best known worst-case gap sequence)<ref>{{cite web |title=Shellsort & Comparisons |url=http://www.cs.wcupa.edu/rkline/ds/shell-comparison.html |access-date=14 November 2015 |archive-date=20 December 2019 |archive-url=https://web.archive.org/web/20191220040546/https://www.cs.wcupa.edu/rkline/ds/shell-comparison.html |url-status=dead }}</ref>
|average-time=depends on gap sequence
|space=О(''n'') total, O(1) auxiliary
|optimal=No
}}
[[File:Shell sorting algorithm color bars.svg|thumb|alt=The steps of Shellsort.|Swapping pairs of items in successive steps of Shellsort with gaps 5, 3, 1]]
'''Shellsort''', also known as '''Shell sort''' or '''Shell's method''', is an [[in-place algorithm|in-place]] [[comparison sort]]. It can be understood as either a generalization of sorting by exchange ([[bubble sort]]) or sorting by insertion ([[insertion sort]]).<ref name="Knuth" /> The method starts by sorting pairs of elements far apart from each other, then progressively reducing the gap between elements to be compared. By starting with far-apart elements, it can move some out-of-place elements into the position faster than a simple nearest-neighbor exchange.
The running time of Shellsort is heavily dependent on the gap sequence it uses. For many practical variants, determining their [[time complexity]] remains an [[open problem]].

The algorithm was first published by [[Donald Shell]] in 1959, and has nothing to do with shells.<ref name="Shell">{{Cite journal
|url=http://penguin.ewu.edu/cscd300/Topic/AdvSorting/p30-shell.pdf
|last=Shell
|first=D. L.
|title=A High-Speed Sorting Procedure
|journal=Communications of the ACM
|volume=2
|issue=7
|year=1959
|pages=30–32
|doi=10.1145/368370.368387
|s2cid=28572656
|access-date=18 October 2011
|archive-date=30 August 2017
|archive-url=https://web.archive.org/web/20170830020037/http://penguin.ewu.edu/cscd300/Topic/AdvSorting/p30-shell.pdf
|url-status=dead
}}</ref><ref>Some older textbooks and references call this the "Shell–Metzner" sort after [[Marlene Metzner Norton]], but according to Metzner, "I had nothing to do with the sort, and my name should never have been attached to it." See {{Cite web |title=Shell sort |url=https://xlinux.nist.gov/dads/HTML/shellsort.html |access-date=2007-07-17 |publisher=National Institute of Standards and Technology}}</ref>

== Description ==
Shellsort is an optimization of [[insertion sort]] that allows the exchange of items that are far apart. The idea is to arrange the list of elements so that, starting anywhere, taking every ''h''th element produces a sorted list. Such a list is said to be ''h''-sorted. It can also be thought of as ''h'' interleaved lists, each individually sorted.<ref name="Sedgewick">
{{Cite book
|last=Sedgewick
|first=Robert
|author-link=Robert Sedgewick (computer scientist)
|title=Algorithms in C
|edition=3rd
|volume=1
|publisher=Addison-Wesley
|year=1998
|pages=[https://archive.org/details/algorithmsinc00sedg/page/273 273–281]
|isbn=978-0-201-31452-6
|url-access=registration
|url=https://archive.org/details/algorithmsinc00sedg/page/273
}}</ref> Beginning with large values of ''h'' allows elements to move long distances in the original list, reducing large amounts of disorder quickly, and leaving less work for smaller ''h''-sort steps to do.<ref name="KR">
{{Cite book
|last1=Kernighan
|first1=Brian W.
|author-link1=Brian Kernighan
|last2=Ritchie
|first2=Dennis M.
|author-link2=Dennis Ritchie
|title=The C Programming Language
|edition=2nd
|publisher=Prentice Hall
|year=1996
|pages=62
|isbn=978-7-302-02412-5
}}</ref> If the list is then ''k-sorted'' for some smaller integer ''k'', then the list remains ''h''-sorted. A final sort with ''h'' = 1 ensures the list is fully sorted at the end,<ref name="Sedgewick"/> but a judiciously chosen decreasing sequence of ''h'' values leaves very little work for this final pass to do.

In simplistic terms, this means if we have an array of 1024 numbers, our first gap (''h'') could be 512. We then run through the list comparing each element in the first half to the element in the second half. Our second gap (''k'') is 256, which breaks the array into four sections (starting at 0, 256, 512, 768), and we make sure the first items in each section are sorted relative to each other, then the second item in each section, and so on. In practice the gap sequence could be anything, but the last gap is always 1 to finish the sort (effectively finishing with an ordinary insertion sort).

An example run of Shellsort with gaps 5, 3 and 1 is shown below.

{|class="wikitable" style="text-align:center"
!
! {{mvar|a}}1 || {{mvar|a}}2 || {{mvar|a}}3 || {{mvar|a}}4
! {{mvar|a}}5 || {{mvar|a}}6 || {{mvar|a}}7 || {{mvar|a}}8
! {{mvar|a}}9 || {{mvar|a}}10 || {{mvar|a}}11 || {{mvar|a}}12
|-
! Input data
| 62 || 83 || 18 || 53 || 07 || 17 || 95 || 86 || 47 || 69 || 25 || 28
|-
! After 5-sorting
| 17 || 28 || 18 || 47 || 07
|bgcolor=lightcyan| 25 ||bgcolor=lightcyan| 83 ||bgcolor=lightcyan| 86 ||bgcolor=lightcyan| 53 ||bgcolor=lightcyan| 69
| 62 || 95
|-
! After 3-sorting
| 17 || 07 || 18
|bgcolor=lightcyan| 47 ||bgcolor=lightcyan| 28 ||bgcolor=lightcyan| 25
| 69 || 62 || 53
|bgcolor=lightcyan| 83 ||bgcolor=lightcyan| 86 ||bgcolor=lightcyan| 95
|-
! After 1-sorting
| 07 ||bgcolor=lightcyan| 17 || 18 ||bgcolor=lightcyan| 25 || 28 ||bgcolor=lightcyan| 47 || 53 ||bgcolor=lightcyan| 62 || 69 ||bgcolor=lightcyan| 83 || 86 ||bgcolor=lightcyan| 95
|}

The first pass, 5-sorting, performs insertion sort on five separate subarrays (''a''1, ''a''6, ''a''11), (''a''2, ''a''7, ''a''12), (''a''3, ''a''8), (''a''4, ''a''9), (''a''5, ''a''10). For instance, it changes the subarray (''a''1, ''a''6, ''a''11) from (62, 17, 25) to (17, 25, 62). The next pass, 3-sorting, performs insertion sort on the three subarrays (''a''1, ''a''4, ''a''7, ''a''10), (''a''2, ''a''5, ''a''8, ''a''11), (''a''3, ''a''6, ''a''9, ''a''12). The last pass, 1-sorting, is an ordinary insertion sort of the entire array (''a''1,..., ''a''12).

As the example illustrates, the subarrays that Shellsort operates on are initially short; later they are longer but almost ordered. In both cases insertion sort works efficiently.

Unlike [[insertion sort]], Shellsort is not a [[sorting algorithm#Stability|stable sort]] since gapped insertions transport equal elements past one another and thus lose their original order. It is an [[Adaptive sort|adaptive sorting algorithm]] in that it executes faster when the input is partially sorted.

== Example ==
This is a [[C Sharp (programming language)|C#]] example using Marcin Ciura's gap sequence, with an inner insertion sort.
<syntaxhighlight lang="csharp">
using System.Collections.Generic;

// Sort an array a[0...n-1].
List<int> gaps = [701, 301, 132, 57, 23, 10, 4, 1]; // Ciura gap sequence

// Start with the largest gap and work down to a gap of 1
// similar to insertion sort but instead of 1, gap is being used in each step
foreach (int gap in gaps)
{
// Do a gapped insertion sort for every element in gaps
// Each loop leaves a[0..gap-1] in gapped order
for (int i = gap; i < n; ++i)
{
// save a[i] in temp and make a hole at position i
int temp = a[i];
// shift earlier gap-sorted elements up until the correct location for a[i] is found
for (int j = i; (j >= gap) && (a[j - gap] > temp); j -= gap)
{
a[j] = a[j - gap];
}
// put temp (the original a[i]) in its correct location
a[j] = temp;
}
}
</syntaxhighlight>

== Gap sequences ==
The question of deciding which gap sequence to use is difficult. Every gap sequence that contains 1 yields a correct sort (as this makes the final pass an ordinary insertion sort); however, the properties of thus obtained versions of Shellsort may be very different. Too few gaps slows down the passes, and too many gaps produces an overhead.

The table below compares most proposed gap sequences published so far. Some of them have decreasing elements that depend on the size of the sorted array (''N''). Others are increasing infinite sequences, whose elements less than ''N'' should be used in reverse order.

{| class="wikitable"
|- style="background-color: #efefef;"
! [[OEIS]]
! General term (''k'' ≥ 1)
! Concrete gaps
! Worst-case time complexity
! Author and year of publication
|----
|
| <math>\left\lfloor\frac{N}{2^k}\right\rfloor</math>
| <math>1, 2, \ldots, \left\lfloor\frac{N}{4}\right\rfloor,
\left\lfloor\frac{N}{2}\right\rfloor
</math>
| <math>\Theta\left(N^2\right)</math> [e.g. when ''N'' = 2''p'']
| [[Donald Shell|Shell]], 1959<ref name="Shell"/>
|----
|
| <math>2 \left\lfloor\frac{N}{2^{k+1}}\right\rfloor + 1</math>
| <math>1, 3, \ldots, \; 2 \left\lfloor\frac{N}{8}\right\rfloor + 1,
\; \; 2 \left\lfloor\frac{N}{4}\right\rfloor + 1
</math>
| <math>\Theta\left(N^\frac{3}{2}\right)</math>
| Frank & Lazarus, 1960<ref>{{Cite journal
|last1=Frank
|first1=R. M.
|last2=Lazarus
|first2=R. B.
|title=A High-Speed Sorting Procedure
|journal=Communications of the ACM
|volume=3
|issue=1
|year=1960
|pages=20–22
|doi=10.1145/366947.366957|s2cid=34066017
|doi-access=free
}}</ref>
|----
| {{OEIS link|A000225}}
| <math>2^k - 1</math>
| <math>1, 3, 7, 15, 31, 63, \ldots</math>
| <math>\Theta\left(N^\frac{3}{2}\right)</math>
| [[Thomas N. Hibbard|Hibbard]], 1963<ref>{{Cite journal
|last=Hibbard
|first=Thomas N.
|title=An Empirical Study of Minimal Storage Sorting
|journal=Communications of the ACM
|volume=6
|issue=5
|year=1963
|pages=206–213
|doi=10.1145/366552.366557|s2cid=12146844
|doi-access=free
}}</ref>
|----
| {{OEIS link|A083318}}
| <math>2^k + 1</math>, prefixed with 1
| <math>1, 3, 5, 9, 17, 33, 65, \ldots</math>
| <math>\Theta\left(N^\frac{3}{2}\right)</math>
| Papernov & Stasevich, 1965<ref>{{Cite journal
|url=http://www.mathnet.ru/links/83f0a81df1ec06f76d3683c6cab7d143/ppi751.pdf
|last1=Papernov
|first1=A. A.
|last2=Stasevich
|first2=G. V.
|title=A Method of Information Sorting in Computer Memories
|journal=Problems of Information Transmission
|volume=1
|issue=3
|year=1965
|pages=63–75}}</ref>
|----
| {{OEIS link|A003586}}
| Successive numbers of the form <math>2^p 3^q</math> ([[3-smooth]] numbers)
| <math>1, 2, 3, 4, 6, 8, 9, 12, \ldots</math>
| <math>\Theta\left(N \log^2 N\right)</math>
| [[Vaughan Ronald Pratt|Pratt]], 1971<ref name="Pratt"/>
|----
| {{OEIS link|A003462}}
| <math>\frac{3^k - 1}{2}</math>, not greater than <math>\left\lceil\frac{N}{3}\right\rceil</math>
| <math>1, 4, 13, 40, 121, \ldots </math>
| <math>\Theta\left(N^\frac{3}{2}\right)</math>
| [[Donald Knuth|Knuth]], 1973,<ref name="Knuth">{{Cite book
|last=Knuth
|first=Donald E. |author-link=Donald Knuth
|title=The Art of Computer Programming. Volume 3: Sorting and Searching
|edition=2nd
|publisher=Addison-Wesley
|location=Reading, Massachusetts
|year=1997
|pages=83–95
|chapter=Shell's method
|isbn=978-0-201-89685-5}}</ref> based on [[Vaughan Ronald Pratt|Pratt]], 1971<ref name="Pratt"/>
|----
| {{OEIS link|A036569}}
| <math>\begin{align}
&\prod\limits_I a_q, \hbox{where} \\
a_0 = {} &3 \\
a_q = {} &\min\left\{n \in \mathbb{N}\colon n \ge \left(\frac{5}{2}\right)^{q+1}, \forall p\colon 0 \le p < q \Rightarrow \gcd(a_p, n) = 1\right\} \\
I = {} &\left\{0 \le q < r \mid q \neq \frac{1}{2}\left(r^2 + r\right) - k \right\} \\
r = {} &\left\lfloor \sqrt{2k + \sqrt{2k}} \right\rfloor
\end{align}</math>
| <math>1, 3, 7, 21, 48, 112, \ldots</math>
| <math>O\left(N^{1 + \sqrt{\frac{8\ln\left(5/2\right)}{\ln(N)}}}\right)</math>
| Incerpi & [[Robert Sedgewick (computer scientist)|Sedgewick]], 1985,<ref>{{Cite journal
|last1=Incerpi
|first1=Janet
|last2=Sedgewick
|first2=Robert |author2-link=Robert Sedgewick (computer scientist)
|title=Improved Upper Bounds on Shellsort
|journal=Journal of Computer and System Sciences
|volume=31
|issue=2
|year=1985
|pages=210–224
|doi=10.1016/0022-0000(85)90042-x|url=https://hal.inria.fr/inria-00076291/file/RR-0267.pdf
}}</ref> [[Donald Knuth|Knuth]]<ref name="Knuth"/>
|----
| {{OEIS link|A036562}}
| <math>4^k + 3 \cdot 2^{k-1} + 1</math>, prefixed with 1
| <math>1, 8, 23, 77, 281, \ldots</math>
| <math>O\left(N^\frac{4}{3}\right)</math>
| Sedgewick, 1982<ref name="Sedgewick"/>
|----
| {{OEIS link|A033622}}
| <math>\begin{cases}
9\left(2^{k} - 2^\frac{k}{2}\right) + 1 & k\text{ even}, \\
8 \cdot 2^{k} - 6 \cdot 2^{(k+1)/2} + 1 & k\text{ odd}
\end{cases}</math>
| <math>1, 5, 19, 41, 109, \ldots</math>
| <math>O\left(N^\frac{4}{3}\right)</math>
| Sedgewick, 1986<ref name="Sedgewick2">
{{Cite journal
|last=Sedgewick
|first=Robert |author-link=Robert Sedgewick (computer scientist)
|title=A New Upper Bound for Shellsort
|journal=Journal of Algorithms
|volume=7
|issue=2
|year=1986
|pages=159–173
|doi=10.1016/0196-6774(86)90001-5
}}</ref>
|----
|
| <math>h_k = \max\left\{\left\lfloor \frac{5h_{k-1}-1}{11} \right\rfloor, 1\right\}, h_0 = N</math>
| <math>1, \ldots, \left\lfloor \frac{5}{11}\left\lfloor \frac{5N-1}{11} \right\rfloor-\frac{1}{11}\right\rfloor,
\left\lfloor \frac{5N-1}{11} \right\rfloor
</math>
| {{unk}}
| [[Gaston Gonnet|Gonnet]] & [[Ricardo Baeza-Yates|{{nowrap|Baeza-Yates}}]], 1991<ref name="Gonnet">{{Cite book
|last1=Gonnet
|first1=Gaston H.
|last2=Baeza-Yates
|first2=Ricardo
|title=Handbook of Algorithms and Data Structures: In Pascal and C
|publisher=Addison-Wesley
|location=Reading, Massachusetts
|edition=2nd
|year=1991
|pages=161–163
|chapter=Shellsort
|isbn=978-0-201-41607-7
|quote=Extensive experiments indicate that the sequence defined by {{math|1=''α'' = 0.45454 < 5/11}} performs significantly better than other sequences. The easiest way to compute {{math|{{floor|0.45454''n''}}}} is by <code>(5 * ''n'' — 1)/11</code> using integer arithmetic.
}}</ref>
|----
| {{OEIS link|A108870}}
| <math>\left\lceil \frac{1}{5} \left(9\cdot \left(\frac{9}{4}\right)^{k-1} - 4 \right) \right\rceil</math> (or equivalently, <math>\left\lceil \frac{\left(9/4\right)^k-1}{\left(9/4\right)-1} \right\rceil</math>)
| <math>1, 4, 9, 20, 46, 103, \ldots</math>
| {{unk}}
| Tokuda, 1992<ref>{{Cite book
|editor-last=van Leeuven
|editor-first=Jan
|chapter=An Improved Shellsort
|last=Tokuda
|first=Naoyuki
|title=Proceedings of the IFIP 12th World Computer Congress on Algorithms, Software, Architecture
|publisher=North-Holland Publishing Co.
|location=Amsterdam
|year=1992
|pages=449–457
|isbn=978-0-444-89747-3}}</ref> (misquote per OEIS)
|----
| {{OEIS link|A102549}}
| Unknown (experimentally derived)
| <math>1, 4, 10, 23, 57, 132, 301, 701</math>
| {{unk}}
| Ciura, 2001<ref name=":0">{{Cite book
|chapter-url=http://sun.aei.polsl.pl/~mciura/publikacje/shellsort.pdf
|archive-url=https://web.archive.org/web/20180923235211/http://sun.aei.polsl.pl/~mciura/publikacje/shellsort.pdf
|archive-date=23 September 2018
|title=Proceedings of the 13th International Symposium on Fundamentals of Computation Theory
|editor-last=Freiwalds
|editor-first=Rusins
|last=Ciura
|first=Marcin
|chapter=Best Increments for the Average Case of Shellsort
|publisher=Springer-Verlag
|location=London
|year=2001
|pages=106–117
|isbn=978-3-540-42487-1}}</ref>
|----
| {{OEIS link|A366726}}
| <math>\left\lceil \frac{\gamma^k-1}{\gamma-1} \right\rceil, \gamma = 2.243609061420001\ldots</math>
| <math>1, 4, 9, 20, 45, 102, 230, 516, \ldots</math>
| {{unk}}
| Lee, 2021<ref>{{cite arXiv
| last = Lee
| first = Ying Wai
| eprint = 2112.11112
| title = Empirically Improved Tokuda Gap Sequence in Shellsort
| class = cs.DS
| date = 21 December 2021
}}</ref>
|----
|
| <math>\left\lfloor 4.0816\cdot 8.5714^{\frac{k}{2.2449}} \right\rfloor</math>
| <math>1, 4, 10, 27, 72, 187, 488, \ldots</math>
| {{unk}}
| Skean, Ehrenborg, Jaromczyk, 2023<ref>{{cite arXiv
| first1 = Oscar
| last1 = Skean
| first2 = Richard
| last2 = Ehrenborg
| first3 = Jerzy W.
| last3 = Jaromczyk
| eprint = 2301.00316
| title = Optimization Perspectives on Shellsort
| class = cs.DS
| date = 1 Jan 2023
}}</ref>
|-
|
|Start at 1. Successive increments are smallest prime <math> \geq 3 </math> times previous
|<math>1, 3, 11, 37, 113, 347, 1049, \ldots</math>
|{{unk}}
|Glenn C. Rhoads<ref>{{Cite web |last=Rhoads |first=Glenn |date=March 1, 2010 |title=Shellsort Increment Sequences |url=https://gcrhoads.byethost4.com/shellSort.html |access-date=May 13, 2025 |website=The Glenn}}</ref>
|}
When the binary representation of ''N'' contains many consecutive zeroes, Shellsort using Shell's original gap sequence makes Θ(''N''2) comparisons in the worst case. For instance, this case occurs for ''N'' equal to a power of two when elements greater and smaller than the median occupy odd and even positions respectively, since they are compared only in the last pass.

Although it has higher complexity than the ''O''(''N'' log ''N'') that is optimal for comparison sorts, Pratt's version lends itself to [[sorting network]]s and has the same asymptotic gate complexity as Batcher's [[bitonic sorter]].

Gonnet and Baeza-Yates observed that Shellsort makes the fewest comparisons on average when the ratios of successive gaps are roughly equal to 2.2.<ref name="Gonnet"/> This is why their sequence with ratio 2.2 and Tokuda's sequence with ratio 2.25 prove efficient. However, it is not known why this is so. Sedgewick recommends using gaps which have low [[greatest common divisor]]s or are pairwise [[coprime]].<ref>{{Cite book
|title=Algorithms in C++, Parts 1–4: Fundamentals, Data Structure, Sorting, Searching
|last=Sedgewick
|first=Robert |author-link=Robert Sedgewick (computer scientist)
|chapter=Shellsort
|publisher=Addison-Wesley
|location=Reading, Massachusetts
|year=1998
|pages=285–292
|isbn=978-0-201-35088-3}}</ref>{{Fv|date=February 2021|reason=There's a lot of discussion of divisibility, but I couldn't find this explicitly stated. E.g. p. 289 says "The increment sequences that we have discussed to this point are effective because successive elements are relatively prime. Another family of increment sequences is effective precisely because successive elements are not relatively prime."}} Gaps which are odd numbers seem to work well in practice: 25% reductions have been observed by avoiding even-numbered gaps. Gaps which avoid multiples of 3 and 5 seem to produce small benefits of < 10%.{{OR|date=May 2023}}

With respect to the average number of comparisons, Ciura's sequence<ref name=":0" /> has the best known performance; gaps greater than 701 were not determined but the sequence can be further extended according to the recursive formula <math>h_k = \lfloor 2.25 h_{k-1} \rfloor</math>.

Tokuda's sequence, defined by the simple formula <math>h_k = \lceil h'_k \rceil</math>, where <math>h'_k = 2.25 h'_{k-1} + 1</math>, <math>h'_1 = 1</math>, can be recommended for practical applications.

If the maximum input size is small, as may occur if Shellsort is used on small subarrays by another recursive sorting algorithm such as [[quicksort]] or [[merge sort]], then it is possible to tabulate an optimal sequence for each input size.<ref>{{cite web
|title=How to choose the lengths of my sub sequences for a shell sort?
|first=Olof |last=Forshell
|date=22 May 2018
|url=https://stackoverflow.com/a/50470237
|website=[[Stack Overflow]]
}} Additional commentary at [https://stackoverflow.com/a/50490873#50490873 Fastest gap sequence for shell sort?] (23 May 2018).</ref><ref>{{cite arXiv
|title=Optimal Gap Sequences in Shellsort for {{math|''n'' ≤ 16}} Elements
|first=Ying Wai |last=Lee
|date=21 December 2021
|eprint=2112.11127
|class=math.CO
}}</ref>

== Computational complexity ==
The following property holds: after ''h''2-sorting of any ''h''1-sorted array, the array remains ''h''1-sorted.<ref>{{Cite journal
|last1=Gale
|first1=David
|author-link=David Gale
|last2=Karp
|first2=Richard M.
|author2-link=Richard M. Karp
|title=A Phenomenon in the Theory of Sorting
|journal=Journal of Computer and System Sciences
|volume=6
|issue=2
|date=April 1972
|pages=103–115
|doi=10.1016/S0022-0000(72)80016-3
|url=https://core.ac.uk/download/pdf/82277625.pdf
|doi-access=free
}}</ref> Every ''h''1-sorted and ''h''2-sorted array is also (''a''1''h''1+''a''2''h''2)-sorted, for any nonnegative integers ''a''1 and ''a''2. The worst-case complexity of Shellsort is therefore connected with the [[coin problem|Frobenius problem]]: for given integers ''h''1,..., ''hn'' with gcd = 1, the Frobenius number ''g''(''h''1,..., ''hn'') is the greatest integer that cannot be represented as ''a''1''h''1+ ... +''anhn'' with nonnegative integer ''a''1,..., ''an''. Using known formulae for Frobenius numbers, we can determine the worst-case complexity of Shellsort for several classes of gap sequences.<ref>{{Cite journal
|last=Selmer
|first=Ernst S.
|author-link=Ernst Sejersted Selmer
|title=On Shellsort and the Frobenius Problem
|journal=BIT Numerical Mathematics
|volume=29
|issue=1
|date=March 1989
|pages=37–40
|doi=10.1007/BF01932703
|hdl=1956/19572
|s2cid=32467267
|hdl-access=free
|url=https://bora.uib.no/bora-xmlui/bitstream/handle/1956/19572/On%20Shellsort%20and%20the%20Frobenius%20problem.pdf
}}</ref> Proven results are shown in the above table.

Mark Allen Weiss proved that Shellsort runs in ''O''(''N'' log ''N'') time when the input array is in reverse order.<ref>{{Cite journal
|last=Weiss
|first=Mark Allen
|title=A good case for Shellsort
|journal=Congressus Numerantium
|volume=73
|date=1989
|pages=59–62
}}</ref>

With respect to the average number of operations, none of the proven results concerns a practical gap sequence. For gaps that are powers of two, Espelid computed this average as <math>0.5349N\sqrt{N}-0.4387N-0.097\sqrt{N}+O(1)</math>.<ref>{{Cite journal
|last=Espelid
|first=Terje O.
|title=Analysis of a Shellsort Algorithm
|journal=BIT Numerical Mathematics
|volume=13
|issue=4
|date=December 1973
|pages=394–400
|doi=10.1007/BF01933401
|s2cid=119443598
}} The quoted result is equation (8) on p. 399.</ref> [[Donald Knuth|Knuth]] determined the average complexity of sorting an ''N''-element array with two gaps (''h'', 1) to be <math>\frac{2N^2}{h} + \sqrt{\pi N^3 h}</math>.<ref name="Knuth" /> It follows that a two-pass Shellsort with ''h'' = Θ(''N''1/3) makes on average ''O''(''N''5/3) comparisons/inversions/running time. [[Andrew Yao|Yao]] found the average complexity of a three-pass Shellsort.<ref>{{Cite journal
|last=Yao
|first=Andrew Chi-Chih
|author-link=Andrew Yao
|title=An Analysis of (''h'', ''k'', 1)-Shellsort
|journal=Journal of Algorithms
|volume=1
|issue=1
|year=1980
|pages=14–50
|doi=10.1016/0196-6774(80)90003-6
|s2cid=3054966
|url=http://pdfs.semanticscholar.org/d569/b8a70a808c6b808ca2e25371c736ce98b14f.pdf
|archive-url=https://web.archive.org/web/20190304043832/http://pdfs.semanticscholar.org/d569/b8a70a808c6b808ca2e25371c736ce98b14f.pdf
|url-status=dead
|archive-date=2019-03-04
|id=STAN-CS-79-726
}}</ref> His result was refined by [[Svante Janson|Janson]] and Knuth:<ref>{{Cite journal
|last1=Janson
|first1=Svante |author1-link=Svante Janson
|last2=Knuth
|first2=Donald E. |author2-link=Donald Knuth
|title=Shellsort with Three Increments
|journal=Random Structures and Algorithms
|volume=10
|issue=1–2
|year=1997
|pages=125–142
|doi=10.1002/(SICI)1098-2418(199701/03)10:1/2<125::AID-RSA6>3.0.CO;2-X
|arxiv=cs/9608105
|citeseerx=10.1.1.54.9911
|url=http://www2.math.uu.se/~svante/papers/sj113.pdf
}}</ref> the average number of comparisons/inversions/running time made during a Shellsort with three gaps (''ch'', ''cg'', 1), where ''h'' and ''g'' are coprime, is <math>\frac{N^2}{4ch} + O(N)</math> in the first pass, <math>\frac{1}{8g}\sqrt{\frac{\pi}{ch}}(h - 1)N^{3/2} + O(hN)</math> in the second pass and <math>\psi(h, g)N + \frac{1}{8}\sqrt{\frac{\pi}{c}}(c - 1)N^{3/2} + O\left((c - 1)gh^{1/2}N\right) + O\left(c^2g^3h^2\right)</math> in the third pass. ''ψ''(''h'', ''g'') in the last formula is a complicated function asymptotically equal to <math>\sqrt{\frac{\pi h}{128}}g + O\left(g^{-1/2}h^{1/2}\right) + O\left(gh^{-1/2}\right)</math>. In particular, when ''h'' = Θ(''N''7/15) and ''g'' = Θ(''N''1/5), the average time of sorting is ''O''(''N''23/15).

Based on experiments, it is conjectured that Shellsort with [[Thomas N. Hibbard|Hibbard]]'s gap sequence runs in ''O''(''N''5/4) average time,<ref name="Knuth" /> and that Gonnet and Baeza-Yates's sequence requires on average 0.41''N'' ln ''N'' (ln ln ''N'' + 1/6) element moves.<ref name="Gonnet" /> Approximations of the average number of operations formerly put forward for other sequences fail when sorted arrays contain millions of elements.

The graph below shows the average number of element comparisons use by various gap sequences, divided by the [[Comparison sort#Number of comparisons required to sort a list|theoretical lower bound]], i.e. log2''N''!. Ciuria's sequence 1, 4, 10, 23, 57, 132, 301, 701 (labelled Ci01) has been extended according to the formula <math>h_k = \lfloor2.25 h_{k-1}\rfloor</math>.

[[File:Shell sort average number of comparisons (English).svg|center]]

Applying the theory of [[Kolmogorov complexity]], Jiang, [[Ming Li|Li]], and [[Paul Vitányi|Vitányi]]
<ref>{{Cite journal
|last1=Jiang
|first1=Tao
|last2=Li
|first2=Ming
|author2-link=Ming Li
|last3=Vitányi
|first3=Paul
|author3-link=Paul Vitányi
|title=A Lower Bound on the Average-Case Complexity of Shellsort
|journal=[[Journal of the ACM]]
|volume=47
|issue=5
|date=September 2000
|pages=905–911
|doi=10.1145/355483.355488
|citeseerx=10.1.1.6.6508
|url=https://homepages.cwi.nl/~paulv/papers/shellsort.pdf
|arxiv=cs/9906008
|s2cid=3265123
}}</ref> proved the following lower bound for the order of the average number of operations/running time in a ''p''-pass Shellsort: Ω(''pN''1+1/''p'') when ''p'' ≤ log2''N'' and Ω(''pN'') when ''p'' > log2''N''.
Therefore, Shellsort has prospects of running in an average time that asymptotically grows like ''N'' log''N'' only when using gap sequences whose number of gaps grows in proportion to the logarithm of the array size. It is, however, unknown whether Shellsort can reach this asymptotic order of average-case complexity, which is optimal for comparison sorts. The lower bound was improved by [[Paul Vitányi|Vitányi]]<ref>{{cite journal
|doi=10.1002/rsa.20737
|last=Vitányi
|first=Paul
|author-link=Paul Vitányi
|date=March 2018
|title=On the average-case complexity of Shellsort
|journal=Random Structures and Algorithms
|volume=52
|issue=2
|pages=354–363
|arxiv=1501.06461
|s2cid=6833808
|url=https://homepages.cwi.nl/~paulv/papers/shell2015.pdf
}}</ref> for every number of passes <math>p</math> to
<math>
\Omega ( N\sum_{k=1}^p h_{k-1}/h_k )
</math>
where <math>h_0=N</math>. This result implies for example the Jiang-Li-Vitányi lower bound for all <math>p</math>-pass increment sequences and improves that lower bound for particular increment sequences. In fact all bounds (lower and upper) currently known for the average case are precisely matched by this lower bound. For example, this gives the new result that the Janson-Knuth upper bound is matched by the resulting lower bound for the used increment sequence, showing that three pass Shellsort for this increment sequence uses <math>\Theta(N^{23/15})</math> comparisons/inversions/running time.
The formula allows us to search for increment sequences that yield lower bounds which are unknown; for example an increment sequence for four passes which has a lower bound greater than
<math>\Omega(pn^{1+1/p}) = \Omega(n^{5/4})</math> for the increment sequence
<math>h_1 = n^{11/16},</math> <math>h_2 = n^{7/16},</math> <math>h_3 = n^{3/16},</math> <math>h_4 = 1</math>. The lower bound becomes
<math>T = \Omega(n\cdot (n^{1-11/16}+n^{11/16-7/16}+n^{7/16-3/16}+n^{3/16}) = \Omega(n^{1+5/16}) = \Omega(n^{21/16}).</math>

The worst-case complexity of any version of Shellsort is of higher order: Plaxton, [[Bjorn Poonen|Poonen]], and [[Torsten Suel|Suel]] showed that it grows at least as rapidly as <math>\Omega\left(N \left( {\log N \over \log \log N} \right)^2\right)</math>.<ref>{{Cite book
|last1=Plaxton
|first1=C. Greg
|last2=Poonen
|first2=Bjorn
|author2-link=Bjorn Poonen
|last3=Suel
|first3=Torsten
|title=Proceedings., 33rd Annual Symposium on Foundations of Computer Science
|chapter=Improved lower bounds for Shellsort
|author3-link=Torsten Suel
|volume=33
|date=24–27 October 1992
|location=Pittsburgh, United States
|pages=226–235
|doi=10.1109/SFCS.1992.267769
|isbn=978-0-8186-2900-6
|citeseerx=10.1.1.43.1393
|s2cid=15095863
|chapter-url=http://engineering.nyu.edu/~suel/papers/shell.pdf
}}</ref><ref>{{Cite journal
|last1=Plaxton
|first1=C. Greg
|last2=Suel
|first2=Torsten
|author2-link=Torsten Suel
|title=Lower Bounds for Shellsort
|journal=Journal of Algorithms
|volume=23 |issue=2
|date=May 1997
|pages=221–240
|doi=10.1006/jagm.1996.0825
|citeseerx=10.1.1.460.2429
|url=http://engineering.nyu.edu/~suel/papers/shell2.pdf
}}</ref>
Robert Cypher proved a stronger lower bound: <math>\Omega\left(N {{(\log N)^2} \over {\log\log N}}\right)</math> when <math>h_{s+1} > h_s</math> for all <math>s</math>.<ref>{{Cite journal
|last=Cypher
|first=Robert
|title=A Lower Bound on the Size of Shellsort Sorting Networks
|journal=SIAM Journal on Computing
|volume=22
|date=1993
|pages=62–71
|doi=10.1137/0222006
}}</ref>

== Applications ==
Shellsort performs more operations and has higher [[CPU cache#Cache miss|cache miss ratio]] than [[quicksort]]. However, since it can be implemented using little code and does not use the [[call stack]], some implementations of the [[qsort]] function in the [[C standard library]] targeted at [[embedded systems]] use it instead of quicksort. Shellsort is, for example, used in the [[uClibc]] library.<ref>{{Cite web
| url=http://git.uclibc.org/uClibc/tree/libc/stdlib/stdlib.c#n700
| title=libc/stdlib/stdlib.c
| first=Manuel III |last=Novoa
| access-date=2014-10-29}}</ref> For similar reasons, in the past, Shellsort was used in the [[Linux kernel]].<ref>{{Cite web
| url=https://github.com/torvalds/linux/blob/72932611b4b05bbd89fafa369d564ac8e449809b/kernel/groups.c#L105
| title=kernel/groups.c
| website=[[GitHub]]
| access-date=2012-05-05}}</ref>

Shellsort can also serve as a sub-algorithm of [[introsort|introspective sort]], to sort short subarrays and to prevent a slowdown when the recursion depth exceeds a given limit. This principle is employed, for instance, in the [[bzip2]] compressor.<ref>{{Cite web
|url=https://www.ncbi.nlm.nih.gov/IEB/ToolBox/CPP_DOC/lxr/source/src/util/compress/bzip2/blocksort.c#L519
|title=bzip2/blocksort.c
|author=Julian Seward
|access-date=2011-03-30}}</ref>

== See also ==
* [[Comb sort]]

== References ==
{{reflist|30em}}

== Bibliography ==
* {{Cite book
|last=Knuth
|first=Donald E. |author-link=Donald Knuth
|title=The Art of Computer Programming. Volume 3: Sorting and Searching
|edition=2nd
|publisher=Addison-Wesley
|location=Reading, Massachusetts
|year=1997
|pages=83–95
|chapter=Shell's method
|isbn=978-0-201-89685-5
|title-link=The Art of Computer Programming }}
* [http://www.cs.princeton.edu/~rs/shell/ Analysis of Shellsort and Related Algorithms], Robert Sedgewick, Fourth European Symposium on Algorithms, Barcelona, September 1996.

== External links ==
{{wikibooks|Algorithm implementation|Sorting/Shell_sort|Shell sort}}
* {{webarchive |url=https://web.archive.org/web/20150310043846/http://www.sorting-algorithms.com/shell-sort |date=10 March 2015 |title=Animated Sorting Algorithms: Shell Sort}} – graphical demonstration
* [https://www.youtube.com/watch?v=CmPA7zE8mx0 Shellsort with gaps 5, 3, 1 as a Hungarian folk dance]

{{sorting}}
{{Use dmy dates|date=April 2020}}

{{Authority control}}

{{DEFAULTSORT:Shellsort}}
[[Category:Comparison sorts]]

Node (computer science)

2025-11-03T18:11:42Z

2605:8D80:5825:1532:D58:188C:5434:D54:

{{Short description|Basic unit of a data structure}}
A '''node''' is a basic unit of a [[data structure]], such as a [[linked list]] or [[Tree (data structure)|tree]] data structure. Nodes contain [[data]] and also may link to other nodes. Links between nodes are often implemented by [[Pointer (computer programming)|pointers]].

[[File:6n-graf.svg|thumb|250px|In [[graph theory]], the image provides a simplified view of a network, where each of the numbers represents a different node.]]

==Nodes and trees ==
[[File:binary tree.svg|right|192px|thumb|A simple [[binary tree]] of size 9 and height 3, with a root node whose value is 2. The above tree is unbalanced and not sorted.]]
Nodes are often arranged into tree structures. A node represents the information contained in a single data structure. These nodes may contain a value or condition, or possibly serve as another independent data structure. Nodes are represented by a single parent node. The highest point on a tree structure is called a root node, which does not have a parent node, but serves as the parent or 'grandparent' of all of the nodes below it in the tree. The height of a node is determined by the total number of edges on the path from that node to the furthest leaf node, and the height of the tree is equal to the height of the root node.<ref>{{cite web|url=http://xlinux.nist.gov/dads//HTML/tree.html|title=tree (data structure)|publisher=[[National Institute of Standards and Technology]]|url-status=live|archive-url=https://web.archive.org/web/20141124223717/http://xlinux.nist.gov/dads/HTML/tree.html|archive-date=2014-11-24}}</ref> Node depth is determined by the distance between that particular node and the root node. The root node is said to have a depth of zero.<ref>{{cite book | last=Teukolsky | first=Roselyn | title=Barron's AP Computer Science A| url=https://archive.org/details/apcomputerscienc0006teuk | url-access=registration |publisher=[[Barron's Educational Series|Barron's]]|isbn = 978-1-4380-0152-4|year=2013 }}</ref> Data can be discovered along these network paths.<ref>{{cite web|url=http://www.eecs.berkeley.edu/~bh/ssch18/trees.html|title=Simply Scheme: Introducing Computer Science ch 18: Trees|publisher=College Of Engineering, [[University of California, Berkeley]]|url-status=live|archive-url=https://web.archive.org/web/20131222183836/http://www.eecs.berkeley.edu/~bh/ssch18/trees.html|archive-date=2013-12-22}}</ref>
An [[IP address]] uses this kind of system of nodes to define its location in a network.

===Definitions===
*'''Child''': A child node is a node extending from another node. For example, a computer with internet access could be considered a child node of a node representing the internet. The inverse relationship is that of a '''parent node'''. If node ''C'' is a child of node ''A'', then ''A'' is the parent node of ''C''.
*'''Degree''': the degree of a node is the number of children of the node.
*'''Depth''': the depth of node ''A'' is the length of the path from ''A'' to the root node. The root node is said to have depth 0.
*'''Edge''': the connection between nodes.
*'''Forest''': a set of trees.
*'''Height''': the height of node ''A'' is the length of the longest path through children to a leaf node.
*'''Internal node''': a node with at least one child.
*'''Leaf node''': a node with no children.
*'''Root node''': a node distinguished from the rest of the tree nodes. Usually, it is depicted as the highest node of the tree.
*'''Sibling nodes''': these are nodes connected to the same parent node.

==Markup languages==
Another common use of node trees is in [[web development]]. In programming, [[XML]] is used to communicate information between computer programmers and computers alike. For this reason XML is used to create common [[communication protocol]]s used in [[office productivity software]], and serves as the base for the development of modern web [[markup language]]s like [[XHTML]]. Though similar in how it is approached by a programmer, [[HTML]] and [[CSS]] is typically the language used to develop website text and design. While XML, HTML and XHTML provide the language and expression, the [[Document Object Model|DOM]] serves as a translator.<ref>{{cite web|url=https://www.w3schools.com/dom/dom_intro.asp|title=XML DOM Introduction|publisher=[[W3Schools]]|url-status=dead|archive-url=https://web.archive.org/web/20140611052725/https://www.w3schools.com/dom/dom_intro.asp|archive-date=2014-06-11|access-date=2018-04-07}}</ref>

=== Node type ===
Different types of nodes in a tree are represented by specific interfaces. In other words, the node type is defined by how it communicates with other nodes. Each node has a node type property, which specifies the type of node, such as sibling or leaf.
For example, if the node type property is the constant properties for a node, this property specifies the type of the node. So if a node type property is the constant node <code>ELEMENT_NODE</code>, one can know that this node object is an object <code>Element</code>. This object uses the <code>Element</code> interface to define all the methods and properties of that particular node.

Different W3C [[World Wide Web Consortium]] node types and descriptions:
* <code>Document</code> represents the entire document (the root-node of the DOM tree)
* <code>DocumentFragment</code> represents a "lightweight" <code>Document</code> object, which can hold a portion of a document
* <code>DocumentType</code> provides an interface to the entities defined for the document
* <code>ProcessingInstruction</code> represents a processing instruction
* <code>EntityReference</code> represents an entity reference
* <code>Element</code> represents an element
* <code>Attr</code> represents an attribute
* <code>Text</code> represents textual content in an element or attribute
* <code>CDATASection</code> represents a [[CDATA]] section in a document (text that will NOT be parsed by a parser)
* <code>Comment</code> represents a comment
* <code>Entity</code> represents an entity
* <code>Notation</code> represents a notation declared in the DTD
{| class="wikitable"
|-
! <code>NodeType</code> !! Named constant
|-
| 1|| <code>ELEMENT_NODE</code>
|-
| 2|| <code>ATTRIBUTE_NODE</code>
|-
| 3|| <code>TEXT_NODE</code>
|-
| 4|| <code>CDATA_SECTION_NODE</code>
|-
| 5|| <code>ENTITY_REFERENCE_NODE</code>
|-
| 6|| <code>ENTITY_NODE</code>
|-
| 7|| <code>PROCESSING_INSTRUCTION_NODE</code>
|-
| 8|| <code>COMMENT_NODE</code>
|-
| 9|| <code>DOCUMENT_NODE</code>
|-
| 10|| <code>DOCUMENT_TYPE_NODE</code>
|-
| 11|| <code>DOCUMENT_FRAGMENT_NODE</code>
|-
| 12|| <code>NOTATION_NODE</code>
|}

=== Node object ===
A node object is represented by a single node in a tree. It can be an element node, attribute node, text node, or any type that is described in section "node type". All objects can inherit properties and methods for dealing with parent and child nodes, but not all of the objects have parent or child nodes. For example, with text nodes that cannot have child nodes, trying to add child nodes results in a [[Document Object Model|DOM]] error.

Objects in the DOM tree may be addressed and manipulated by using methods on the objects. The public interface of a DOM is specified in its [[application programming interface]] (API). The history of the Document Object Model is intertwined with the history of the "[[browser wars]]" of the late 1990s between [[Netscape Navigator]] and [[Microsoft Internet Explorer]], as well as with that of [[JavaScript]] and [[JScript]], the first [[scripting language]]s to be widely implemented in the [[Browser engine|layout engine]]s of [[web browser]]s.

==See also==

* [[Vertex (graph theory)]]

==References==
{{Reflist}}

==External links==
* [http://www.community-of-knowledge.de/beitrag/data-trees-as-a-means-of-presenting-complex-data-analysis/ Data Trees as a Means of Presenting Complex Data Analysis] by Sally Knipe
* [http://tree.phi-sci.com STL-like C++ tree class] {{Webarchive|url=https://web.archive.org/web/20201126210313/http://tree.phi-sci.com/ |date=2020-11-26 }}
* [https://web.archive.org/web/20180306223544/http://ideainfo.8m.com/ Description of tree data structures from ideainfo.8m.com]
* [http://wormweb.org/celllineage WormWeb.org: Interactive Visualization of the ''C. elegans'' Cell Tree] - Visualize the entire cell lineage tree of the nematode ''C. elegans'' (javascript)

[[Category:Linked lists]]
[[Category:Graph data structures]]

Automatic differentiation

2025-11-03T16:21:49Z

2605:8D80:5825:1532:D58:188C:5434:D54: /* C++ */

{{Short description|Numerical calculations carrying along derivatives}}
In [[mathematics]] and [[computer algebra]], '''automatic differentiation''' ('''auto-differentiation''', '''autodiff''', or '''AD'''), also called '''algorithmic differentiation''', '''computational differentiation''', and '''differentiation arithmetic'''<ref>{{cite journal|last=Neidinger|first=Richard D.|title=Introduction to Automatic Differentiation and MATLAB Object-Oriented Programming|journal=SIAM Review| year=2010| volume=52| issue=3| pages=545–563| url=http://academics.davidson.edu/math/neidinger/SIAMRev74362.pdf|doi=10.1137/080743627| citeseerx=10.1.1.362.6580|s2cid=17134969 }}</ref><ref name="baydin2018automatic">{{cite journal|last1=Baydin|first1=Atilim Gunes|last2=Pearlmutter| first2=Barak|last3=Radul|first3=Alexey Andreyevich|last4=Siskind|first4=Jeffrey|title=Automatic differentiation in machine learning: a survey| journal=Journal of Machine Learning Research|year=2018|volume=18|pages=1–43|url=http://jmlr.org/papers/v18/17-468.html}}</ref><ref name="Dawood.Megahed.2023">[[Hend Dawood]] and [[Nefertiti Megahed]] (2023). Automatic differentiation of uncertainties: an interval computational differentiation for first and higher derivatives with implementation. PeerJ Computer Science 9:e1301 https://doi.org/10.7717/peerj-cs.1301.</ref><ref name="Dawood.Megahed.2019">[[Hend Dawood]] and [[Nefertiti Megahed]] (2019). A Consistent and Categorical Axiomatization of Differentiation Arithmetic Applicable to First and Higher Order Derivatives. Punjab University Journal of Mathematics. 51(11). pp. 77-100. doi: 10.5281/zenodo.3479546. http://doi.org/10.5281/zenodo.3479546.</ref> is a set of techniques to evaluate the [[partial derivative]] of a function specified by a computer program. Automatic differentiation is a subtle and central tool to automate the simultaneous computation of the numerical values of arbitrarily complex functions and their derivatives with no need for the symbolic representation of the derivative, only the function rule or an algorithm thereof is required.<ref name="Dawood.Megahed.2023"/><ref name="Dawood.Megahed.2019"/> Auto-differentiation is thus neither numeric nor symbolic, nor is it a combination of both. It is also preferable to ordinary numerical methods: In contrast to the more traditional numerical methods based on finite differences, auto-differentiation is 'in theory' exact, and in comparison to symbolic algorithms, it is computationally inexpensive.<ref name="Dawood-attribution">{{Creative Commons text attribution notice|cc=bysa4|url=https://pmc.ncbi.nlm.nih.gov/articles/PMC10280627/|author(s)=Dawood and Megahed}}</ref><ref name="Dawood.Megahed.2023"/><ref name="Dawood.Dawood.2022">[[Hend Dawood]] and [[Yasser Dawood]] (2022). Interval Root Finding and Interval Polynomials: Methods and Applications in Science and Engineering. In S. Chakraverty, editor, Polynomial Paradigms: Trends and Applications in Science and Engineering, chapter 15. IOP Publishing. ISBN 978-0-7503-5065-5. doi: 10.1088/978-0-7503-5067-9ch15. URL https://doi.org/10.1088/978-0-7503-5067-9ch15.</ref>

Automatic differentiation exploits the fact that every computer calculation, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions ([[exponential function|exp]], [[natural logarithm|log]], [[sine|sin]], [[cosine|cos]], etc.). By applying the [[chain rule]] repeatedly to these operations, partial derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor of more arithmetic operations than the original program.

== Difference from other differentiation methods ==
[[Image:AutomaticDifferentiationNutshell.png|right|thumb|300px|Figure 1: How automatic differentiation relates to symbolic differentiation]]
Automatic differentiation is distinct from [[symbolic differentiation]] and [[numerical differentiation]].
Symbolic differentiation faces the difficulty of converting a computer program into a single [[mathematical expression]] and can lead to inefficient code. Numerical differentiation (the method of finite differences) can introduce [[round-off error]]s in the [[discretization]] process and cancellation. Both of these classical methods have problems with calculating higher derivatives, where complexity and errors increase. Finally, both of these classical methods are slow at computing partial derivatives of a function with respect to ''many'' inputs, as is needed for [[gradient descent|gradient]]-based [[Optimization (mathematics)|optimization]] algorithms. Automatic differentiation solves all of these problems.

== Applications ==
Currently, for its efficiency and accuracy in computing first and higher order [[derivative]]s, auto-differentiation is a celebrated technique with diverse applications in [[scientific computing]] and [[mathematics]]. It should therefore come as no surprise that there are numerous computational implementations of auto-differentiation. Among these, one mentions [[INTLAB]], Sollya, and InCLosure.<ref name="Rump.1999">[[Siegfried M. Rump]] (1999). INTLAB–INTerval LABoratory. In T. Csendes, editor, Developments in Reliable Computing, pages 77–104. Kluwer Academic Publishers, Dordrecht.</ref><ref name="Chevillard.Joldes.Lauter.2010">S. Chevillard, M. Joldes, and C. Lauter. Sollya (2010). An Environment for the Development of Numerical Codes. In K. Fukuda, J. van der Hoeven, M. Joswig, and N. Takayama, editors, Mathematical Software - ICMS 2010, volume 6327 of Lecture Notes in Computer Science, pages 28–31, Heidelberg, Germany. Springer.</ref><ref name="Dawood.Inc4.2022">[[Hend Dawood]] (2022). [[InCLosure]] (Interval enCLosure)–A Language and Environment for Reliable Scientific Computing. Computer Software, Version 4.0. Department of Mathematics. Faculty of Science, Cairo University, Giza, Egypt, September 2022. url: https://doi.org/10.5281/zenodo.2702404.</ref> In practice, there are two types (modes) of algorithmic differentiation: a forward-type and a reversed-type.<ref name="Dawood.Megahed.2023" /><ref name="Dawood.Megahed.2019" /> Presently, the two types are highly correlated and complementary and both have a wide variety of applications in, e.g., non-linear [[optimization]], [[sensitivity analysis]], [[robotics]], [[machine learning]], [[computer graphics]], and [[computer vision]].<ref name="Dawood-attribution"/><ref name="Fries.2019">Christian P. Fries (2019). Stochastic Automatic Differentiation: Automatic Differentiation for Monte-Carlo Simulations. Quantitative Finance, 19(6):1043–1059. doi: 10.1080/14697688.2018.1556398. url: https://doi.org/10.1080/14697688.2018.1556398.</ref><ref name="Dawood.Megahed.2023"/><ref name="Dawood.Megahed.2019"/><ref name="Dawood.Dawood.2020">[[Hend Dawood]] and [[Yasser Dawood]] (2020). Universal Intervals: Towards a Dependency-Aware Interval Algebra. In S. Chakraverty, editor, Mathematical Methods in Interdisciplinary Sciences. chapter 10, pages 167–214. John Wiley & Sons, Hoboken, New Jersey. ISBN 978-1-119-58550-3. doi: 10.1002/9781119585640.ch10. url: https://doi.org/10.1002/9781119585640.ch10.</ref><ref name="Dawood.2014">[[Hend Dawood]] (2014). Interval Mathematics as a Potential Weapon against Uncertainty. In S. Chakraverty, editor, Mathematics of Uncertainty Modeling in the Analysis of Engineering and Science Problems. chapter 1, pages 1–38. IGI Global, Hershey, PA. ISBN 978-1-4666-4991-0.</ref> Automatic differentiation is particularly important in the field of [[machine learning]]. For example, it allows one to implement [[backpropagation]] in a [[neural network (machine learning)|neural network]] without a manually-computed derivative.

== Forward and reverse accumulation ==
=== Chain rule of partial derivatives of composite functions ===
Fundamental to automatic differentiation is the decomposition of differentials provided by the [[chain rule]] of [[partial derivative]]s of [[function composition|composite functions]]. For the simple composition
<math display="block">\begin{align}
y &= f(g(h(x))) = f(g(h(w_0))) = f(g(w_1)) = f(w_2) = w_3 \\
w_0 &= x \\
w_1 &= h(w_0) \\
w_2 &= g(w_1) \\
w_3 &= f(w_2) = y
\end{align}</math>
the chain rule gives
<math display="block">\frac{\partial y}{\partial x} = \frac{\partial y}{\partial w_2} \frac{\partial w_2}{\partial w_1} \frac{\partial w_1}{\partial x} = \frac{\partial f(w_2)}{\partial w_2} \frac{\partial g(w_1)}{\partial w_1} \frac{\partial h(w_0)}{\partial x}</math>

=== Two types of automatic differentiation ===
Usually, two distinct modes of automatic differentiation are presented.
* '''forward accumulation''' (also called '''bottom-up''', '''forward mode''', or '''tangent mode''')
* '''reverse accumulation''' (also called '''top-down''', '''reverse mode''', or '''adjoint mode''')
Forward accumulation specifies that one traverses the chain rule from inside to outside (that is, first compute <math>\partial w_1/ \partial x</math> and then <math>\partial w_2/\partial w_1</math> and lastly <math>\partial y/\partial w_2</math>), while reverse accumulation traverses from outside to inside (first compute <math>\partial y/\partial w_2</math> and then <math>\partial w_2/\partial w_1</math> and lastly <math>\partial w_1/\partial x</math>). More succinctly,
* Forward accumulation computes the recursive relation: <math>\frac{\partial w_i}{\partial x} = \frac{\partial w_i}{\partial w_{i-1}} \frac{\partial w_{i-1}}{\partial x}</math> with <math>w_3 = y</math>, and,
* Reverse accumulation computes the recursive relation: <math>\frac{\partial y}{\partial w_i} = \frac{\partial y}{\partial w_{i+1}} \frac{\partial w_{i+1}}{\partial w_{i}}</math> with <math>w_0 = x</math>.

The value of the partial derivative, called the ''seed'', is propagated forward or backward and is initially <math>\frac{\partial x}{\partial x}=1</math> or <math>\frac{\partial y}{\partial y}=1</math>. Forward accumulation evaluates the function and calculates the derivative with respect to one independent variable in one pass. For each independent variable <math>x_1,x_2,\dots,x_n</math> a separate pass is therefore necessary in which the derivative with respect to that independent variable is set to one (<math>\frac{\partial x_1}{\partial x_1}=1</math>) and of all others to zero (<math>\frac{\partial x_2}{\partial x_1}= \dots = \frac{\partial x_n}{\partial x_1} = 0</math>). In contrast, reverse accumulation requires the evaluated partial functions for the partial derivatives. Reverse accumulation therefore evaluates the function first and calculates the derivatives with respect to all independent variables in an additional pass.

Which of these two types should be used depends on the sweep count. The [[Computational complexity theory|computational complexity]] of one sweep is proportional to the complexity of the original code.
* Forward accumulation is more efficient than reverse accumulation for functions {{math|''f'' : '''R'''''n'' → '''R'''''m''}} with {{math|''n'' ≪ ''m''}} as only {{math|''n''}} sweeps are necessary, compared to {{math|''m''}} sweeps for reverse accumulation.
* Reverse accumulation is more efficient than forward accumulation for functions {{math|''f'' : '''R'''''n'' → '''R'''''m''}} with {{math|''n'' ≫ ''m''}} as only {{math|''m''}} sweeps are necessary, compared to {{math|''n''}} sweeps for forward accumulation.
[[Backpropagation]] of errors in multilayer perceptrons, a technique used in [[machine learning]], is a special case of reverse accumulation.<ref name="baydin2018automatic" />

Forward accumulation was introduced by R.E. Wengert in 1964.<ref name="Wengert1964"/> According to Andreas Griewank, reverse accumulation has been suggested since the late 1960s, but the inventor is unknown.<ref name="grie2012">{{cite book |last=Griewank |first=Andreas |title=Optimization Stories |chapter=Who invented the reverse mode of differentiation? |year=2012 |series=Documenta Mathematica Series |volume= 6|pages=389–400 |doi=10.4171/dms/6/38 |doi-access=free |isbn=978-3-936609-58-5 |chapter-url=https://ftp.gwdg.de/pub/misc/EMIS/journals/DMJDMV/vol-ismp/52_griewank-andreas-b.pdf }}</ref> [[Seppo Linnainmaa]] published reverse accumulation in 1976.<ref name="lin1976">{{cite journal |last=Linnainmaa |first=Seppo |year=1976 |title=Taylor Expansion of the Accumulated Rounding Error |journal=BIT Numerical Mathematics |volume=16 |issue=2 |pages=146–160 |doi=10.1007/BF01931367 |s2cid=122357351 }}</ref>

=== Forward accumulation ===
[[File:ForwardAD.png|thumb|Forward accumulation]]
In forward accumulation AD, one first fixes the ''independent variable'' with respect to which differentiation is performed and computes the derivative of each sub-[[expression (mathematics)|expression]] recursively. In a pen-and-paper calculation, this involves repeatedly substituting the derivative of the ''inner'' functions in the chain rule:
<math display="block">\begin{align}
\frac{\partial y}{\partial x}
&= \frac{\partial y}{\partial w_{n-1}} \frac{\partial w_{n-1}}{\partial x} \\[6pt]
&= \frac{\partial y}{\partial w_{n-1}} \left(\frac{\partial w_{n-1}}{\partial w_{n-2}} \frac{\partial w_{n-2}}{\partial x}\right) \\[6pt]
&= \frac{\partial y}{\partial w_{n-1}} \left(\frac{\partial w_{n-1}}{\partial w_{n-2}} \left(\frac{\partial w_{n-2}}{\partial w_{n-3}} \frac{\partial w_{n-3}}{\partial x}\right)\right) \\[6pt]
&= \cdots
\end{align}</math>
This can be generalized to multiple variables as a matrix product of [[Jacobian matrix and determinant|Jacobian]]s.

Compared to reverse accumulation, forward accumulation is natural and easy to implement as the flow of derivative information coincides with the order of evaluation. Each variable <math>w_i</math> is augmented with its derivative <math>\dot w_i</math> (stored as a numerical value, not a symbolic expression),
<math display="block">\dot w_i = \frac{\partial w_i}{\partial x}</math>
as denoted by the dot. The derivatives are then computed in sync with the evaluation steps and combined with other derivatives via the chain rule.

Using the chain rule, if <math>w_i</math> has predecessors in the computational graph:
:<math>\dot w_i = \sum_{j \in \{\text{predecessors of i}\}} \frac{\partial w_i}{\partial w_j} \dot w_j</math>

[[Image:ForwardAccumulationAutomaticDifferentiation.png|right|thumb|300px|Figure 2: Example of forward accumulation with computational graph]]

As an example, consider the function:
<math display="block">\begin{align}
y
&= f(x_1, x_2) \\
&= x_1 x_2 + \sin x_1 \\
&= w_1 w_2 + \sin w_1 \\
&= w_3 + w_4 \\
&= w_5
\end{align}</math>
For clarity, the individual sub-expressions have been labeled with the variables <math>w_i</math>.

The choice of the independent variable to which differentiation is performed affects the ''seed'' values {{math|''ẇ''1}} and {{math|''ẇ''2}}. Given interest in the derivative of this function with respect to {{math|''x''1}}, the seed values should be set to:
<math display="block">\begin{align}
\dot w_1 = \frac{\partial w_1}{\partial x_1} = \frac{\partial x_1}{\partial x_1} = 1 \\
\dot w_2 = \frac{\partial w_2}{\partial x_1} = \frac{\partial x_2}{\partial x_1} = 0
\end{align}</math>

With the seed values set, the values propagate using the chain rule as shown. Figure 2 shows a pictorial depiction of this process as a computational graph.
:{| class="wikitable"
!Operations to compute value !!Operations to compute derivative
|-
|<math>w_1 = x_1</math> || <math>\dot w_1 = 1</math> (seed)
|-
|<math>w_2 = x_2</math> || <math>\dot w_2 = 0</math> (seed)
|-
|<math>w_3 = w_1 \cdot w_2</math> || <math>\dot w_3 = w_2 \cdot \dot w_1 + w_1 \cdot \dot w_2</math>
|-
|<math>w_4 = \sin w_1</math> || <math>\dot w_4 = \cos w_1 \cdot \dot w_1</math>
|-
|<math>w_5 = w_3 + w_4</math> || <math>\dot w_5 = \dot w_3 + \dot w_4</math>
|}

To compute the [[gradient]] of this example function, which requires not only <math>\tfrac{\partial y}{\partial x_1}</math> but also <math>\tfrac{\partial y}{\partial x_2}</math>, an ''additional'' sweep is performed over the computational graph using the seed values <math>\dot w_1 = 0; \dot w_2 = 1</math>.

==== Implementation ====
===== Pseudocode =====
Forward accumulation calculates the function and the derivative (but only for one independent variable each) in one pass. The associated method call expects the expression ''Z'' to be derived with regard to a variable ''V''. The method returns a pair of the evaluated function and its derivative. The method traverses the expression tree recursively until a variable is reached. If the derivative with respect to this variable is requested, its derivative is 1, 0 otherwise. Then the partial function as well as the partial derivative are evaluated.<ref name=demm22>{{cite book|author = Maximilian E. Schüle, Maximilian Springer, [[Alfons Kemper]], [[Thomas Neumann]] |title=Proceedings of the Sixth Workshop on Data Management for End-To-End Machine Learning |chapter=LLVM code optimisation for automatic differentiation |date=2022|pages=1–4 |doi = 10.1145/3533028.3533302|isbn=9781450393751 |s2cid=248853034 |language=English}}</ref>
<syntaxhighlight lang="cpp">
Tuple<float, float> evaluateAndDerive(Expression Z, Variable V) {
if isVariable(Z) {
if (Z = V) {
return {valueOf(Z), 1};
} else {
return {valueOf(Z), 0};
}
} else if (Z = A + B) {
{a, a'} = evaluateAndDerive(A, V);
{b, b'} = evaluateAndDerive(B, V);
return {a + b, a' + b'};
} else if (Z = A - B) {
{a, a'} = evaluateAndDerive(A, V);
{b, b'} = evaluateAndDerive(B, V);
return {a - b, a' - b'};
} else if (Z = A * B) {
{a, a'} = evaluateAndDerive(A, V);
{b, b'} = evaluateAndDerive(B, V);
return {a * b, b * a' + a * b'};
}
}
</syntaxhighlight>

===== C++ =====
<syntaxhighlight lang="cpp">
import std;

using std::unique_ptr;

struct ValueAndPartial {
float value;
float partial;
};

class Expression {
public:
virtual ValueAndPartial evaluateAndDerive(class Variable* variable) const noexcept = 0;
virtual ~Expression() = default;
};

class Variable : public Expression {
private:
float value;
public:
explicit Variable(float value):
value{value} {}

[[nodiscard]]
ValueAndPartial evaluateAndDerive(Variable* variable) const noexcept override {
float partial = (this == variable) ? 1.0f : 0.0f;
return {value, partial};
}

[[nodiscard]]
float getValue() const noexcept {
return value;
}
};

class Plus : public Expression {
private:
unique_ptr<Expression> a;
unique_ptr<Expression> b;
public:
Plus(unique_ptr<Expression> a, unique_ptr<Expression> b):
a{std::move(a)}, b{std::move(b)} {}

[[nodiscard]]
ValueAndPartial evaluateAndDerive(Variable* variable) const noexcept override {
auto [valueA, partialA] = a->evaluateAndDerive(variable);
auto [valueB, partialB] = b->evaluateAndDerive(variable);
return {valueA + valueB, partialA + partialB};
}
};

class Multiply : public Expression {
private:
unique_ptr<Expression> a;
unique_ptr<Expression> b;
public:
Multiply(unique_ptr<Expression> a, unique_ptr<Expression> b):
a{std::move(a)}, b{std::move(b)} {}

[[nodiscard]]
ValueAndPartial evaluateAndDerive(Variable* variable) const noexcept override {
auto [valueA, partialA] = a->evaluateAndDerive(variable);
auto [valueB, partialB] = b->evaluateAndDerive(variable);
return {valueA * valueB, valueB * partialA + valueA * partialB};
}
};

int main() {
// Example: Finding the partials of z = x * (x + y) + y * y at (x, y) = (2, 3)
Variable x(2.0f);
Variable y(3.0f);

// Create expressions using smart pointers
unique_ptr<Plus> p1 = std::make_unique<Plus>(std::make_unique<Variable>(x), std::make_unique<Variable>(y));
unique_ptr<Multiply> m1 = std::make_unique<Multiply>(std::make_unique<Variable>(x), std::move(p1));
unique_ptr<Multiply> m2 = std::make_unique<Multiply>(std::make_unique<Variable>(y), std::make_unique<Variable>(y));
unique_ptr<Plus> z = std::make_unique<Plus>(std::move(m1), std::move(m2));

// Evaluate the partials
float xPartial = z->evaluateAndDerive(&x).partial;
float yPartial = z->evaluateAndDerive(&y).partial;

std::println("∂z/∂x = {}, ∂z/∂y = {}", xPartial, yPartial);
// Output: ∂z/∂x = 7, ∂z/∂y = 8
return 0;
}
</syntaxhighlight>

=== Reverse accumulation ===
[[File:AutoDiff.webp|thumb|Reverse accumulation]]
In reverse accumulation AD, the ''dependent variable'' to be differentiated is fixed and the derivative is computed ''with respect to'' each sub-[[expression (mathematics)|expression]] recursively. In a pen-and-paper calculation, the derivative of the ''outer'' functions is repeatedly substituted in the chain rule:
<math display="block">\begin{align}
\frac{\partial y}{\partial x}
&= \frac{\partial y}{\partial w_1} \frac{\partial w_1}{\partial x}\\
&= \left(\frac{\partial y}{\partial w_2} \frac{\partial w_2}{\partial w_1}\right) \frac{\partial w_1}{\partial x}\\
&= \left(\left(\frac{\partial y}{\partial w_3} \frac{\partial w_3}{\partial w_2}\right) \frac{\partial w_2}{\partial w_1}\right) \frac{\partial w_1}{\partial x}\\
&= \cdots
\end{align}</math>

In reverse accumulation, the quantity of interest is the ''adjoint'', denoted with a bar <math>\bar w_i</math>; it is a derivative of a chosen dependent variable with respect to a subexpression <math>w_i</math>:
<math display="block">\bar w_i = \frac{\partial y}{\partial w_i}</math>

Using the chain rule, if <math>w_i</math> has successors in the computational graph:
:<math>\bar w_i = \sum_{j \in \{\text{successors of i}\}} \bar w_j \frac{\partial w_j}{\partial w_i}</math>

Reverse accumulation traverses the chain rule from outside to inside, or in the case of the computational graph in Figure 3, from top to bottom. The example function is scalar-valued, and thus there is only one seed for the derivative computation, and only one sweep of the computational graph is needed to calculate the (two-component) gradient. This is only [[space–time tradeoff|half the work]] when compared to forward accumulation, but reverse accumulation requires the storage of the intermediate variables {{math|''w''''i''}} as well as the instructions that produced them in a data structure known as a "tape" or a Wengert list<ref>{{cite journal|last1=Bartholomew-Biggs|first1=Michael| last2=Brown|first2=Steven|last3=Christianson|first3=Bruce|last4=Dixon|first4=Laurence|date=2000|title=Automatic differentiation of algorithms|journal=Journal of Computational and Applied Mathematics| volume=124|issue=1–2|pages=171–190| doi=10.1016/S0377-0427(00)00422-2|bibcode=2000JCoAM.124..171B|hdl=2299/3010|hdl-access=free}}</ref> (however, Wengert published forward accumulation, not reverse accumulation<ref name="Wengert1964">{{cite journal|author=R.E. Wengert|title=A simple automatic derivative evaluation program|journal=Comm. ACM|volume=7
|issue=8|year=1964|pages=463–464|doi=10.1145/355586.364791|s2cid=24039274|doi-access=free}}</ref>), which may consume significant memory if the computational graph is large. This can be mitigated to some extent by storing only a subset of the intermediate variables and then reconstructing the necessary work variables by repeating the evaluations, a technique known as [[rematerialization]]. [[checkpointing scheme|Checkpointing]] is also used to save intermediary states.

[[Image:ReverseaccumulationAD.png|right|thumb|300px|Figure 3: Example of reverse accumulation with computational graph]]

The operations to compute the derivative using reverse accumulation are shown in the table below (note the reversed order):
{{block indent|
; Operations to compute derivative
:<math>\bar w_5 = 1 \text{ (seed)}</math>
:<math>\bar w_4 = \bar w_5 \cdot 1</math>
:<math>\bar w_3 = \bar w_5 \cdot 1</math>
:<math>\bar w_2 = \bar w_3 \cdot w_1</math>
:<math>\bar w_1 = \bar w_3 \cdot w_2 + \bar w_4 \cdot \cos w_1</math>
}}
The data flow graph of a computation can be manipulated to calculate the gradient of its original calculation. This is done by adding an adjoint node for each primal node, connected by adjoint edges which parallel the primal edges but flow in the opposite direction. The nodes in the adjoint graph represent multiplication by the derivatives of the functions calculated by the nodes in the primal. For instance, addition in the primal causes fanout in the adjoint; fanout in the primal causes addition in the adjoint;{{efn|In terms of weight matrices, the adjoint is the [[transpose]]. Addition is the [[covector]] <math>[1 \cdots 1]</math>, since <math>[1 \cdots 1]\left[\begin{smallmatrix}x_1 \\ \vdots \\ x_n \end{smallmatrix}\right] = x_1 + \cdots + x_n,</math> and fanout is the vector <math>\left[\begin{smallmatrix}1 \\ \vdots \\ 1 \end{smallmatrix}\right],</math> since <math>\left[\begin{smallmatrix}1 \\ \vdots \\ 1 \end{smallmatrix}\right][x] = \left[\begin{smallmatrix}x \\ \vdots \\ x \end{smallmatrix}\right].</math>}} a [[unary operation|unary]] function {{math|1=''y'' = ''f''(''x'')}} in the primal causes {{math|1=''x̄'' = ''ȳ'' ''f''′(''x'')}} in the adjoint; etc.

==== Implementation ====
===== Pseudo code =====
Reverse accumulation requires two passes: In the forward pass, the function is evaluated first and the partial results are cached. In the reverse pass, the partial derivatives are calculated and the previously derived value is backpropagated. The corresponding method call expects the expression ''Z'' to be derived and ''seeded'' with the derived value of the parent expression. For the top expression, Z differentiated with respect to Z, this is 1. The method traverses the expression tree recursively until a variable is reached and adds the current ''seed'' value to the derivative expression.<ref name=ssdbm21>{{cite book|author= Maximilian E. Schüle, Harald Lang, Maximilian Springer, [[Alfons Kemper]], [[Thomas Neumann]], Stephan Günnemann|title=33rd International Conference on Scientific and Statistical Database Management |chapter=In-Database Machine Learning with SQL on GPUs |date=2021|pages=25–36 |doi = 10.1145/3468791.3468840|isbn=9781450384131 |s2cid=235386969 |language=English}}</ref><ref name=dpd>{{cite journal|author= Maximilian E. Schüle, Harald Lang, Maximilian Springer, [[Alfons Kemper]], [[Thomas Neumann]], Stephan Günnemann|title=Recursive SQL and GPU-support for in-database machine learning|journal=Distributed and Parallel Databases|date=2022|volume=40 |issue=2–3 |pages=205–259 |doi = 10.1007/s10619-022-07417-7|s2cid=250412395 |language=English|doi-access=free}}</ref>
<syntaxhighlight lang="cpp">
void derive(Expression Z, float seed) {
if isVariable(Z) {
partialDerivativeOf(Z) += seed;
} else if (Z = A + B) {
derive(A, seed);
derive(B, seed);
} else if (Z = A - B) {
derive(A, seed);
derive(B, -seed);
} else if (Z = A * B) {
derive(A, valueOf(B) * seed);
derive(B, valueOf(A) * seed);
}
}
</syntaxhighlight>

===== C++ =====
<syntaxhighlight lang="cpp">
import std;

using std::unique_ptr;

class Expression {
private:
float value;
public:
virtual void evaluate() noexcept = 0;
virtual void derive(float seed) noexcept = 0;
virtual ~Expression() = default;

[[nodiscard]]
float getValue() const noexcept {
return value;
}
};

class Variable : public Expression {
private:
float partial;
public:
explicit Variable(float value):
value{value}, partial{0.0f} {}

void evaluate() noexcept override {
// no-op
}

void derive(float seed) noexcept override {
partial += seed;
}

[[nodiscard]]
float getPartial() const noexcept {
return partial;
}
};

class Plus : public Expression {
private:
unique_ptr<Expression> a;
unique_ptr<Expression> b;
public:
Plus(unique_ptr<Expression> a, unique_ptr<Expression> b):
a{std::move(a)}, b{std::move(b)} {}

void evaluate() noexcept override {
a->evaluate();
b->evaluate();
value = a->value + b->value;
}

void derive(float seed) noexcept override {
a->derive(seed);
b->derive(seed);
}
};

class Multiply : public Expression {
private:
unique_ptr<Expression> a;
unique_ptr<Expression> b;
public:
Multiply(unique_ptr<Expression> a, unique_ptr<Expression> b):
a{std::move(a)}, b{std::move(b)} {}

void evaluate() noexcept override {
a->evaluate();
b->evaluate();
value = a->value * b->value;
}

void derive(float seed) noexcept override {
a->derive(b->value * seed);
b->derive(a->value * seed);
}
};

int main() {
// Example: Finding the partials of z = x * (x + y) + y * y at (x, y) = (2, 3)
Variable x(2.0f);
Variable y(3.0f);

// Create expressions using smart pointers
unique_ptr<Plus> p1 = std::make_unique<Plus>(std::make_unique<Variable>(x), std::make_unique<Variable>(y));
unique_ptr<Multiply> m1 = std::make_unique<Multiply>(std::make_unique<Variable>(x), std::move(p1));
unique_ptr<Multiply> m2 = std::make_unique<Multiply>(std::make_unique<Variable>(y), std::make_unique<Variable>(y));
unique_ptr<Plus> z = std::make_unique<Plus>(std::move(m1), std::move(m2));

// Evaluate the expression
z->evaluate();
std::println("z = {}", z->getValue());
// Output: z = 19

// Derivative calculation (partials)
z->derive(1); // Derivative with respect to x and y

std::println("∂z/∂x = {}, ∂z/∂y = {}", x.getPartial(), y.getPartial());
// Output: ∂z/∂x = 7, ∂z/∂y = 8

return 0;
}
</syntaxhighlight>

=== Beyond forward and reverse accumulation ===

Forward and reverse accumulation are just two (extreme) ways of traversing the chain rule. The problem of computing a full Jacobian of {{math|''f'' : '''R'''''n'' → '''R'''''m''}} with a minimum number of arithmetic operations is known as the ''optimal Jacobian accumulation'' (OJA) problem, which is [[NP-complete]].<ref>{{Cite journal|first=Uwe|last=Naumann|journal=Mathematical Programming|volume=112|issue=2|pages=427–441|date=April 2008|doi=10.1007/s10107-006-0042-z|title=Optimal Jacobian accumulation is NP-complete|citeseerx=10.1.1.320.5665|s2cid=30219572}}</ref> Central to this proof is the idea that algebraic dependencies may exist between the local partials that label the edges of the graph. In particular, two or more edge labels may be recognized as equal. The complexity of the problem is still open if it is assumed that all edge labels are unique and algebraically independent.

== Automatic differentiation using dual numbers ==

Forward mode automatic differentiation is accomplished by augmenting the [[Algebra over a field|algebra]] of [[real numbers]] and obtaining a new [[arithmetic]]. An additional component is added to every number to represent the derivative of a function at the number, and all arithmetic operators are extended for the augmented algebra. The augmented algebra is the algebra of [[dual numbers]].

Replace every number <math>\,x</math> with the number <math>x + x'\varepsilon</math>, where <math>x'</math> is a real number, but <math>\varepsilon</math> is an [[abstract number]] with the property <math>\varepsilon^2=0</math> (an [[infinitesimal]]; see ''[[Smooth infinitesimal analysis]]''). Using only this, regular arithmetic gives
<math display="block">\begin{align}
(x + x'\varepsilon) + (y + y'\varepsilon) &= x + y + (x' + y')\varepsilon \\
(x + x'\varepsilon) - (y + y'\varepsilon) &= x - y + (x' - y')\varepsilon \\
(x + x'\varepsilon) \cdot (y + y'\varepsilon) &= xy + xy'\varepsilon + yx'\varepsilon + x'y'\varepsilon^2 = xy + (x y' + yx')\varepsilon \\
(x + x'\varepsilon) / (y + y'\varepsilon) &= (x/y + x'\varepsilon/y) / (1 + y'\varepsilon/y) = (x/y + x'\varepsilon/y) \cdot (1 - y'\varepsilon/y) = x/y + (x'/y - xy'/y^2)\varepsilon
\end{align}</math>
using <math>(1 + y'\varepsilon/y) \cdot (1 - y'\varepsilon/y) = 1</math>.

Now, [[polynomials]] can be calculated in this augmented arithmetic. If <math>P(x) = p_0 + p_1 x + p_2x^2 + \cdots + p_n x^n</math>, then
<math display="block">\begin{align}
P(x + x'\varepsilon) &= p_0 + p_1(x + x'\varepsilon) + \cdots + p_n (x + x'\varepsilon)^n \\
&= p_0 + p_1 x + \cdots + p_n x^n + p_1x'\varepsilon + 2p_2xx'\varepsilon + \cdots + np_n x^{n-1} x'\varepsilon \\
&= P(x) + P^{(1)}(x)x'\varepsilon
\end{align}</math>
where <math>P^{(1)}</math> denotes the derivative of <math>P</math> with respect to its first argument, and <math>x'</math>, called a ''seed'', can be chosen arbitrarily.

The new arithmetic consists of [[ordered pair]]s, elements written <math>\langle x, x' \rangle</math>, with ordinary arithmetics on the first component, and first order differentiation arithmetic on the second component, as described above. Extending the above results on polynomials to [[analytic functions]] gives a list of the basic arithmetic and some standard functions for the new arithmetic:
<math display="block">\begin{align}
\left\langle u,u'\right\rangle + \left\langle v,v'\right\rangle &= \left\langle u + v, u' + v' \right\rangle \\
\left\langle u,u'\right\rangle - \left\langle v,v'\right\rangle &= \left\langle u - v, u' - v' \right\rangle \\
\left\langle u,u'\right\rangle * \left\langle v,v'\right\rangle &= \left\langle u v, u'v + uv' \right\rangle \\
\left\langle u,u'\right\rangle / \left\langle v,v'\right\rangle &= \left\langle \frac{u}{v}, \frac{u'v - uv'}{v^2} \right\rangle \quad ( v\ne 0) \\
\sin\left\langle u,u'\right\rangle &= \left\langle \sin(u) , u' \cos(u) \right\rangle \\
\cos\left\langle u,u'\right\rangle &= \left\langle \cos(u) , -u' \sin(u) \right\rangle \\
\exp\left\langle u,u'\right\rangle &= \left\langle \exp u , u' \exp u \right\rangle \\
\log\left\langle u,u'\right\rangle &= \left\langle \log(u) , u'/u \right\rangle \quad (u>0) \\
\left\langle u,u'\right\rangle^k &= \left\langle u^k , u' k u^{k - 1} \right\rangle \quad (u \ne 0) \\
\left| \left\langle u,u'\right\rangle \right| &= \left\langle \left| u \right| , u' \operatorname{sign} u \right\rangle \quad (u \ne 0)
\end{align}</math>
and in general for the primitive function <math>g</math>,
<math display="block">g(\langle u,u' \rangle , \langle v,v' \rangle ) = \langle g(u,v) , g_u(u,v) u' + g_v(u,v) v' \rangle</math>
where <math>g_u</math> and <math>g_v</math> are the derivatives of <math>g</math> with respect to its first and second arguments, respectively.

When a binary basic arithmetic operation is applied to mixed arguments—the pair <math>\langle u, u' \rangle</math> and the real number <math>c</math>—the real number is first lifted to <math>\langle c, 0 \rangle</math>. The derivative of a function <math>f : \R\to\R</math> at the point <math>x_0</math> is now found by calculating <math>f(\langle x_0, 1 \rangle)</math> using the above arithmetic, which gives <math>\langle f ( x_0 ) , f' ( x_0 ) \rangle </math> as the result.

=== Implementation ===
An example implementation based on the dual number approach follows.

==== Pseudo code ====
<syntaxhighlight lang="cpp">
Dual plus(Dual A, Dual B) {
return {
realPartOf(A) + realPartOf(B),
infinitesimalPartOf(A) + infinitesimalPartOf(B)
};
}

Dual minus(Dual A, Dual B) {
return {
realPartOf(A) - realPartOf(B),
infinitesimalPartOf(A) - infinitesimalPartOf(B)
};
}

Dual multiply(Dual A, Dual B) {
return {
realPartOf(A) * realPartOf(B),
realPartOf(B) * infinitesimalPartOf(A) + realPartOf(A) * infinitesimalPartOf(B)
};
}

X = {x, 0};
Y = {y, 0};
Epsilon = {0, 1};
xPartial = infinitesimalPartOf(f(X + Epsilon, Y));
yPartial = infinitesimalPartOf(f(X, Y + Epsilon));
</syntaxhighlight>

==== C++ ====
<syntaxhighlight lang="cpp">
import std;

class Dual {
private:
float realPart;
float infinitesimalPart;
public:
explicit Dual(float realPart, float infinitesimalPart = 0.0f):
realPart{realPart}, infinitesimalPart{infinitesimalPart} {}

Dual operator+(const Dual& other) noexcept {
return Dual(
realPart + other.realPart,
infinitesimalPart + other.infinitesimalPart
);
}

Dual operator*(const Dual& other) noexcept {
return Dual(
realPart * other.realPart,
other.realPart * infinitesimalPart + realPart * other.infinitesimalPart
);
}

[[nodiscard]]
float getRealPart() const noexcept {
return realPart;
}

[[nodiscard]]
float getInfinitesimalPart() const noexcept {
return infinitesimalPart;
}
};

// Example: Finding the partials of z = x * (x + y) + y * y at (x, y) = (2, 3)
[[nodiscard]]
Dual f(const Dual& x, const Dual& y) noexcept {
return x * (x + y) + y * y;
}

int main() {
Dual x(2.0f);
Dual y(3.0f);
Dual epsilon(0.0f, 1.0f);
Dual a = f(x + epsilon, y);
Dual b = f(x, y + epsilon);
std::println("∂z/∂x = {}, ∂z/∂y = {}", a.getInfinitesimalPart(), b.getInfinitesimalPart());
// Output: ∂z/∂x = 7, ∂z/∂y = 8
return 0;
}
</syntaxhighlight>

===Vector arguments and functions===

Multivariate functions can be handled with the same efficiency and mechanisms as univariate functions by adopting a directional derivative operator. That is, if it is sufficient to compute <math>y' = \nabla f(x)\cdot x'</math>, the directional derivative <math>y' \in \R^m</math> of <math>f:\R^n\to\R^m</math> at <math>x \in \R^n</math> in the direction <math>x' \in \R^n</math> may be calculated as <math>(\langle y_1,y'_1\rangle, \ldots, \langle y_m,y'_m\rangle) = f(\langle x_1,x'_1\rangle, \ldots, \langle x_n,x'_n\rangle)</math> using the same arithmetic as above. If all the elements of <math>\nabla f</math> are desired, then <math>n</math> function evaluations are required. Note that in many optimization applications, the directional derivative is indeed sufficient.

===High order and many variables===

The above arithmetic can be generalized to calculate second order and higher derivatives of multivariate functions. However, the arithmetic rules quickly grow complicated: complexity is quadratic in the highest derivative degree. Instead, truncated [[Taylor series|Taylor polynomial]] algebra can be used. The resulting arithmetic, defined on generalized dual numbers, allows efficient computation using functions as if they were a data type. Once the Taylor polynomial of a function is known, the derivatives are easily extracted.

== Implementation ==

Forward-mode AD is implemented by a [[nonstandard interpretation]] of the program in which real numbers are replaced by dual numbers, constants are lifted to dual numbers with a zero epsilon coefficient, and the numeric primitives are lifted to operate on dual numbers. This nonstandard interpretation is generally implemented using one of two strategies: ''source code transformation'' or ''operator overloading''.

=== Source code transformation (SCT) ===
[[Image:SourceTransformationAutomaticDifferentiation.png|thumb|right|300px|Figure 4: Example of how source code transformation could work]]

The source code for a function is replaced by an automatically generated source code that includes statements for calculating the derivatives interleaved with the original instructions.

[[Source code transformation]] can be implemented for all programming languages, and it is also easier for the compiler to do compile time optimizations. However, the implementation of the AD tool itself is more difficult and the build system is more complex.

=== Operator overloading (OO) ===

[[Image:OperatorOverloadingAutomaticDifferentiation.png|thumb|right|300px|Figure 5: Example of how operator overloading could work]]
[[Operator overloading]] is a possibility for source code written in a language supporting it. Objects for real numbers and elementary mathematical operations must be overloaded to cater for the augmented arithmetic depicted above. This requires no change in the form or sequence of operations in the original source code for the function to be differentiated, but often requires changes in basic data types for numbers and vectors to support overloading and often also involves the insertion of special flagging operations. Due to the inherent operator overloading overhead on each loop, this approach usually demonstrates weaker speed performance.

=== Operator overloading and source code transformation ===
Overloaded Operators can be used to extract the valuation graph, followed by automatic generation of the AD-version of the primal function at run-time. Unlike the classic OO AAD{{Clarify|date=July 2025}}, such AD-function does not change from one iteration to the next one. Hence there is any OO or tape interpretation run-time overhead per Xi sample.

With the AD-function being generated at runtime, it can be optimised to take into account the current state of the program and precompute certain values. In addition, it can be generated in a way to consistently utilize native CPU vectorization to process 4(8)-double chunks of user data (AVX2\AVX512 speed up x4-x8). With multithreading added into account, such approach can lead to a final acceleration of order 8 × #Cores compared to the traditional AAD tools. A reference implementation is available on GitHub.<ref>{{Cite web|url=https://github.com/matlogica/aadc-prototype|title=AADC Prototype Library|date=June 22, 2022|via=GitHub}}</ref>

==See also==
* [[Differentiable programming]]

==Notes==
{{Notelist}}

==References==
{{Reflist|30em}}

== Further reading ==
* {{cite book
| last = Rall
| first = Louis B.
| title = Automatic Differentiation: Techniques and Applications
| publisher = [[Springer Science+Business Media|Springer]]
| series = Lecture Notes in Computer Science
| volume = 120
| year = 1981
| isbn = 978-3-540-10861-0
}}
* {{cite book
|last1 = Griewank
|first1 = Andreas
|last2 = Walther
|first2 = Andrea | author2-link = Andrea Walther
|title = Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation
|edition = 2nd
|publisher = [[Society for Industrial and Applied Mathematics|SIAM]]
|series = Other Titles in Applied Mathematics
|volume = 105
|year = 2008
|doi = 10.1137/1.9780898717761
|isbn = 978-0-89871-659-7
|url = https://epubs.siam.org/doi/book/10.1137/1.9780898717761 
}}
* {{cite journal
|last=Neidinger
|first=Richard
|title=Introduction to Automatic Differentiation and MATLAB Object-Oriented Programming
|journal=SIAM Review
|year=2010|volume=52
|issue=3
|pages=545–563
|doi=10.1137/080743627
|url=http://academics.davidson.edu/math/neidinger/SIAMRev74362.pdf
|access-date=2013-03-15
|citeseerx=10.1.1.362.6580
|s2cid=17134969
}}
* {{cite book
| last = Naumann
| first = Uwe
| title = The Art of Differentiating Computer Programs
| publisher = [[Society for Industrial and Applied Mathematics|SIAM]]
| series = Software-Environments-tools
| year = 2012
| isbn = 978-1-611972-06-1
}}
* {{cite book
| last = Henrard
| first = Marc
| title = Algorithmic Differentiation in Finance Explained
| publisher = [[Palgrave Macmillan]]
| series = Financial Engineering Explained
| year = 2017
| isbn = 978-3-319-53978-2
}}

==External links==
* [http://www.autodiff.org/ www.autodiff.org], An "entry site to everything you want to know about automatic differentiation"
* [http://www.autodiff.org/?module=Applications&application=HC1 Automatic Differentiation of Parallel OpenMP Programs]
* [https://www.researchgate.net/publication/241730000_Automatic_Differentiation_C_Templates_and_Photogrammetry Automatic Differentiation, C++ Templates and Photogrammetry]
* [https://web.archive.org/web/20070927120356/http://www.vivlabs.com/subpage_ad.php Automatic Differentiation, Operator Overloading Approach]
* [http://tapenade.inria.fr:8080/tapenade/index.jsp Compute analytic derivatives of any Fortran77, Fortran95, or C program through a web-based interface] Automatic Differentiation of Fortran programs
* [http://www.win-vector.com/dfiles/AutomaticDifferentiationWithScala.pdf Description and example code for forward Automatic Differentiation in Scala] {{Webarchive|url=https://web.archive.org/web/20160803214549/http://www.win-vector.com/dfiles/AutomaticDifferentiationWithScala.pdf |date=2016-08-03 }}
* [https://www.finmath.net/finmath-lib/concepts/stochasticautomaticdifferentiation/ finmath-lib stochastic automatic differentiation], Automatic differentiation for random variables (Java implementation of the stochastic automatic differentiation).
* [https://web.archive.org/web/20140423121504/http://developers.opengamma.com/quantitative-research/Adjoint-Algorithmic-Differentiation-OpenGamma.pdf Adjoint Algorithmic Differentiation: Calibration and Implicit Function Theorem]
* [http://www.quantandfinancial.com/2017/02/automatic-differentiation-templated.html C++ Template-based automatic differentiation article] and [https://github.com/omartinsky/QuantAndFinancial/tree/master/autodiff_templated implementation]
* [https://github.com/google/tangent Tangent] [https://research.googleblog.com/2017/11/tangent-source-to-source-debuggable.html Source-to-Source Debuggable Derivatives]
* [http://www.nag.co.uk/doc/techrep/pdf/tr5_10.pdf Exact First- and Second-Order Greeks by Algorithmic Differentiation]
* [http://www.nag.co.uk/Market/articles/adjoint-algorithmic-differentiation-of-gpu-accelerated-app.pdf Adjoint Algorithmic Differentiation of a GPU Accelerated Application]
* [http://www.nag.co.uk/Market/seminars/Uwe_AD_Slides_July13.pdf Adjoint Methods in Computational Finance Software Tool Support for Algorithmic Differentiationop]
* [https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/xva-pricing-application-financial-services-white-papers.pdf More than a Thousand Fold Speed Up for xVA Pricing Calculations with Intel Xeon Scalable Processors]
* [https://github.com/ExcessPhase/ctaylor Sparse truncated Taylor series implementation with VBIC95 example for higher order derivatives]

{{Differentiable computing}}
{{Authority control}}

{{DEFAULTSORT:Automatic Differentiation}}
[[Category:Differential calculus]]
[[Category:Computer algebra]]
[[Category:Articles with example pseudocode]]
[[Category:Articles with example Python (programming language) code]]
[[Category:Articles with example C++ code]]

Pointer swizzling

2025-11-03T07:08:34Z

2605:8D80:5825:1532:D58:188C:5434:D54: /* Example */

{{Short description|Computer science term}}
{{No footnotes|date=May 2011}}
{{Use dmy dates|date=December 2021|cs1-dates=y}}
In [[computer science]], '''pointer swizzling''' is the conversion of references based on name or [[offset (computer science)|position]] into direct [[pointer (computer programming)|pointer]] references ([[memory address]]es). It is typically performed during [[deserialization]] or [[loader (computing)|loading]] of a relocatable object from a disk file, such as an [[executable file]] or pointer-based [[data structure]].

The reverse operation, replacing memory pointers with position-independent symbols or positions, is sometimes referred to as '''unswizzling''', and is performed during [[serialization]] (saving). Alternatively, both operations can also be referred to as swizzling.

==Example==
It is easy to create a [[linked list]] data structure using elements like this:

<syntaxhighlight lang=c>
struct LinkedList {
int data;
struct LinkedList* next;
};
</syntaxhighlight>

But saving the list to a file and then reloading it will (on most operating systems) break every link and render the list useless because the nodes will almost never be loaded into the same memory locations. One way to usefully save and retrieve the list is to assign a unique id number to each node and then '''unswizzle''' the pointers by turning them into a field indicating the id number of the next node:

<syntaxhighlight lang=c>
struct UnswizzledLinkedList {
int data;
unsigned int id;
unsigned int id_of_next_node;
};
</syntaxhighlight>

Records like these can be saved to a file in any order and reloaded without breaking the list. Other options include saving the file offset of the next node or a number indicating its position in the sequence of saved records, or simply saving the nodes in-order to the file.

After loading such a list, finding a node based on its number is cumbersome and inefficient (serial search). Traversing the list was very fast with the original "next" pointers. To convert the list back to its original form, or '''swizzle''' the pointers, requires finding the address of each node and turning the <code>id_of_next_node</code> fields back into direct pointers to the right node.

==Methods of unswizzling==
There are a potentially unlimited number of forms into which a pointer can be unswizzled, but some of the most popular include:
* The offset of the pointed-to object in the file
* The index of the pointed-to object in some sequence of records
* A unique identifier possessed by the pointed-to object, such as a person's [[Social Security number]]; in databases, all pointers are unswizzled in this manner (see [[Foreign key]]).

==Methods of swizzling==
Swizzling in the general case can be complicated. The reference [[graph (abstract data type)|graph]] of pointers might contain an arbitrary number of [[cycle (graph theory)|cycle]]s; this complicates maintaining a mapping from the old unswizzled values to the new addresses. [[Associative array]]s are useful for maintaining the mapping, while algorithms such as [[breadth-first search]] help to traverse the graph, although both of these require extra storage. Various [[serialization]] [[library (computing)|libraries]] provide general swizzling systems. In many cases, however, swizzling can be performed with simplifying assumptions, such as a [[tree (data structure)|tree]] or [[list (abstract data type)|list]] structure of references.

The different types of swizzling are:
* Automatic swizzling 
* On-demand swizzling 

==Potential security weaknesses==
For security, unswizzling and swizzling must be implemented with great caution. In particular, an attacker's presentation of a specially crafted file may allow access to addresses outside of the expected and proper bounds. In systems with weak memory protection this can lead to exposure of confidential data or modification of code likely to be executed. If the system does not implement guards against execution of data the system may be severely compromised by the installation of various kinds of [[malware]].

Methods of protection include verifications prior to releasing the data to an application:
* That every offset lies within the bounds of the data read.
* That a table of indexes and the records pointed to is similarly constrained.
* That identifiers are unique and, if sensitive, encrypted.
* That all variable-length data is restrained to lengths not exceeding the actual allocation.
* That allocations are of reasonable size.
* That allocations made that are not loaded with data read are cleared, or loaded with some specific pattern.
{{Expand list|date=August 2011}}

==See also==
* [[Relocation (computing)|Relocation]], an eager form of pointer modification

==References==
{{Reflist|refs=}}

==Further reading==
* {{cite news |title=Pointer swizzling at page fault time: efficiently supporting huge address spaces on standard hardware |author-first=Paul R. |author-last=Wilson |newspaper=ACM SIGARCH Computer Architecture News |volume=19 |issue=4 |pages=6–13 |date=1991-07-01 |orig-date=June 1991 |doi=10.1145/122576.122577 |url=}}
* {{cite journal |title=Adaptable Pointer Swizzling Strategies in Object Bases: Design, Realization, and Quantitative Analysis |author-first1=Alfons |author-last1=Kemper |author-first2=Donald |author-last2=Kossmann |journal=The International Journal on Very Large Data Bases |volume=4 |issue=3 |pages=519–567 |date=July 1995 |doi=10.1007/BF01231646 |s2cid=4556203 |url=http://www.vldb.org/journal/VLDBJ4/P519.pdf |access-date=2021-12-08 |url-status=live |archive-url=https://web.archive.org/web/20080725153221/http://www.vldb.org/journal/VLDBJ4/P519.pdf |archive-date=2008-07-25}} (49 pages)
* {{cite book |author-first=Derek |author-last=Crawford |title=Derek's ABC of C |volume=2 |pages=340–343 |date=June 1992}}

[[Category:Memory management]]
[[Category:Pointers (computer programming)]]