Building standalone Rmarkdown documents using the data language engine

Base64

Base64 encoding is a widely-used, standard encoding to be used for all non-sensitive binary data. It is based on translating 6-bits of information into one of 64 alphanumeric and symbolic characters.

Encoding in base64 using the data_encode function works as follows:

b64 = knitrdata::data_encode("test.RDS","base64")

By default this function will silently return the encoded data as a character string. This character string can then be visualized using the cat function so that it can be copied and pasted directly into a data chunk:

cat(b64)
#> H4sIAAAAAAAAA4vgYmBgYGZgAWJmViCTgTU0xE3XAigmDOQwAfE7KM0IVMQJpPmS
#> 83MLEpNL4jPzSopTC9FkWZISi1OhYrxgcQj9D6QTZJUDBwMY2H9ApaFqBKDOAZnF
#> mAhjJMEYyUAbkK1jzUvMTS2G6mPCog9NeXJOYjFMOUyQKyWxJFEvrQhoEppyzqL8
#> cj2YDSBfMDUAif////8FuRYAeDNrwDsBAAA=

This is only practical for relatively small data files, so for larger files, one can place the output in a file:

data_encode("test.RDS","base64",output="test.RDS.base64")
cat(readLines("test.RDS.base64"),sep="\n")
#> H4sIAAAAAAAAA4vgYmBgYGZgAWJmViCTgTU0xE3XAigmDOQwAfE7KM0IVMQJpPmS
#> 83MLEpNL4jPzSopTC9FkWZISi1OhYrxgcQj9D6QTZJUDBwMY2H9ApaFqBKDOAZnF
#> mAhjJMEYyUAbkK1jzUvMTS2G6mPCog9NeXJOYjFMOUyQKyWxJFEvrQhoEppyzqL8
#> cj2YDSBfMDUAif////8FuRYAeDNrwDsBAAA=

Though it is rarely necessary to call the data_decode function directly when working with the data chunks in Rmarkdown documents, base64 encoded data can be decoded as follows:

rds = data_decode(b64,"base64",as_text=FALSE)
writeBin(rds,"test_output.RDS")
y = readRDS("test_output.RDS")
y
#>   a b
#> 1 1 a
#> 2 2 b
#> 3 3 c

GPG

Ecryption of data using GPG requires a properly configured GPG keyring. The functioning of GPG and GPG keyrings is beyond the scope of this document, but numerous websites explain how GPG works and how to install a GPG keyring, including the main gpg website.

For the purposes of this vignette, I will generate a test GPG private-public key pair using the gpg package, however in real use scenarios proper keys would typically be generated using the gpg command line tool (or an equivalent alternative) with appropriate options.

id = gpg::gpg_keygen("test","test@test.org")

Next one uses this key to encode a data file:

enc = data_encode("test.RDS","gpg",options = list(receiver=id))
cat(enc)

Note that the ID of the desired encryption key must be supplied as the receiver in the options list input argument.

Decoding works as follows:

rds = data_decode(enc,"gpg")
writeBin(rds,"test_output.RDS")
y = readRDS("test_output.RDS")
y

Note that there is no need to supply the receiver ID when decoding because the appropriate private key is in the keyring. When decoding, the gpg package or keyring may prompt for a password to unlock the decryption key if the key is password protected.

We can delete the public-private key pair we created for this exercise from our keyring as follows:

gpg::gpg_delete(id,secret=TRUE)

Text data chunks

The simplest possible data chunk is a text data chunk containing plain text. Textual data can be directly placed into a data chunk in an Rmarkdown document as follows:

```{data output.var="d"}
a,b
1,a
2,b
3,c
```

When the Rmarkdown document is knitted, this chunk will put the text contents of the chunk into the variable d, which will then contain the chunk contents as a character string. For the example CSV data above, the character string can then be converted into a data.frame using read.csv:

read.csv(text=d)
#>   a b
#> 1 1 a
#> 2 2 b
#> 3 3 c

One can also load the data directly into a data.frame using the loader.function chunk option. The loader.function should be a function (or a character string containing the name of a function) whose first input argument will be the name of a file. A file containing the (decoded) data chunk contents will be passed to this function and the output will be assigned to the variable name contained in output.var.

```{data output.var="d",loader.function=read.csv}
a,b
1,a
2,b
3,c
```

This will assign to d the output of read.csv applied to the CSV data in the chunk.

d
#>   a b
#> 1 1 a
#> 2 2 b
#> 3 3 c

Additional input arguments can be passed to loader.function by supplying a list as the loader.ops chunk option:

```{data output.var="d",loader.function=read.csv,loader.ops=list(header=FALSE)}
a,b
1,a
2,b
3,c
```

d
#>   V1 V2
#> 1  a  b
#> 2  1  a
#> 3  2  b
#> 4  3  c

Note that in this case the first line of the CSV data has been treated as data instead of as a header because we supplied the header=FALSE optional argument.

Text documents in data chunks

The data inside a text data chunk does not have to be scientific data. It can be any textual information, including the contents of formatting files used by Rmarkdown to generate final output documents. These include BibTeX files with references, LaTeX style files (.cls) and bibliography style files (.csl). For example, if we include the following in a Rmarkdown document:

```{data output.file="references.bib",echo=FALSE}
@article{MeynardTestingmethodsspecies2019,
  ids = {MeynardTestingmethodsspecie,MeynardTestingmethodsspeciesinpress},
  title = {Testing Methods in Species Distribution Modelling Using Virtual Species: What Have We Learnt and What Are We Missing?},
  shorttitle = {Testing Methods in Species Distribution Modelling Using Virtual Species},
  author = {Meynard, Christine N. and Leroy, Boris and Kaplan, David M.},
  year = {2019},
  month = dec,
  volume = {42},
  pages = {2021--2036},
  issn = {0906-7590, 1600-0587},
  doi = {10.1111/ecog.04385},
  file = {/home/dmk/papers/meynard.et.al.2019.testing_methods_in_species_distribution_modelling_using_virtual_species.pdf},
  journal = {Ecography},
  keywords = {artificial species,environmental niche models,niche,simulations,species distribution modelling,virtual ecologist},
  language = {en},
  number = {12}
}

@article{SantosConsequencesdriftcarcass2018,
  title = {Consequences of Drift and Carcass Decomposition for Estimating Sea Turtle Mortality Hotspots},
  author = {Santos, Bianca S. and Kaplan, David M. and Friedrichs, Marjorie A. M. and Barco, Susan G. and Mansfield, Katherine L. and Manning, James P.},
  year = {2018},
  month = jan,
  volume = {84},
  pages = {319--336},
  issn = {1470-160X},
  doi = {10.1016/j.ecolind.2017.08.064},
  copyright = {All rights reserved},
  file = {/home/dmk/papers/santos.et.al.2018.consequences_of_drift_and_carcass_decomposition_for_estimating_sea_turtle.pdf},
  journal = {Ecological Indicators},
  keywords = {Carcass decomposition,Chesapeake bay,Conservation,Drift leeway,Drift simulations,Endangered species,Sea turtle mortality,Sea turtle strandings}
}
```

This will generate the file references.bib from the chunk contents. Note that one uses output.file instead of output.var to save the contents to a file. Textual output to a file can also be achieved using the cat language environment as described in the R Markdown Cookbook here, though the data language engine provides more options for the handling of chunk contents (for example, one can use base64 encoding to embed Rmarkdown documents within Rmarkdown documents).

As style files like the BibTeX file described above are only used by knitr/pandoc in the final formatting phase of generating an output document, these files can be generated from data chunks during the initial phases of the knitting. This allows them to be stored inside the Rmarkdown document itself, with no need for the external file prior to knitting.

Base64-encoded binary data chunks

Base64 encoded binary data is incorporated into a chunk by copying the output of the data_encode function into the chunk and supplying the format="binary" chunk option:

```{data output.var="b",format="binary",echo=FALSE}
H4sIAAAAAAAAA4vgYmBgYGZgZgNiViCTgTU0xE3XAigmDOQwAfE7KM3IwMLACaT5
kvNzCxKTS+Iz80qKUwvRZFmSEotToWK8YHEI/Q+kE2SVAwcDGNh/QKUhaph5wc6B
6GOCsFmQzWfLSS1LzSkGsgTAshBRxkQYIwnGSEbTyJqck1gM0wc3LQ3okfwiiN2o
yvMSc1NhypmwWANWzvIfzUCulMSSRL20IqBmNAM5i/LL9WCGgnzJ1AAk/v///xdk
OwBVQunahwEAAA==
```

This will place the decoded contents of the chunk into a raw vector b. Note that format must be specified as 'binary' and that we have chosen echo=FALSE to avoid including lots of ugly encoded content in our formatted document. By default, when format="binary", it is assumed that encoding="base64", but this can also be supplied as a chunk option for additional clarity.

The contents of the raw vector b must be written to a file before they can be read back into the Rmarkdown session:

writeBin(b,"test_output.RDS")

We can combine the decoding and the writing steps by specifying the output.file chunk option instead of output.var:

```{data output.file="test_output.RDS",format="binary",echo=FALSE}
H4sIAAAAAAAAA4vgYmBgYGZgZgNiViCTgTU0xE3XAigmDOQwAfE7KM3IwMLACaT5
kvNzCxKTS+Iz80qKUwvRZFmSEotToWK8YHEI/Q+kE2SVAwcDGNh/QKUhaph5wc6B
6GOCsFmQzWfLSS1LzSkGsgTAshBRxkQYIwnGSEbTyJqck1gM0wc3LQ3okfwiiN2o
yvMSc1NhypmwWANWzvIfzUCulMSSRL20IqBmNAM5i/LL9WCGgnzJ1AAk/v///xdk
OwBVQunahwEAAA==
```

This will save the decoded data to the filename given by output.file. Then this file can be read back into the Rmarkdown session:

readRDS("test_output.RDS")
#>   a b
#> 1 1 a
#> 2 2 b
#> 3 3 c

Finally, we can combine all three steps (decoding, writing to disk, reading back into R) using the loader.function chunk option:

```{data output.var="b",format="binary",echo=FALSE,loader.function=readRDS}
H4sIAAAAAAAAA4vgYmBgYGZgZgNiViCTgTU0xE3XAigmDOQwAfE7KM3IwMLACaT5
kvNzCxKTS+Iz80qKUwvRZFmSEotToWK8YHEI/Q+kE2SVAwcDGNh/QKUhaph5wc6B
6GOCsFmQzWfLSS1LzSkGsgTAshBRxkQYIwnGSEbTyJqck1gM0wc3LQ3okfwiiN2o
yvMSc1NhypmwWANWzvIfzUCulMSSRL20IqBmNAM5i/LL9WCGgnzJ1AAk/v///xdk
OwBVQunahwEAAA==
```

Given these options, during knitting, the contents of the data chunk will be decoded, written to a temporary file as binary data and then read back into the R session using the readRDS function. In the end, the variable whose name is given by output.var will be assigned the output of readRDS:

b
#>   a b
#> 1 1 a
#> 2 2 b
#> 3 3 c

GPG-encoded data chunks

GPG chunks work similar to base64 chunks except that one must specify encoding="gpg". To demonstrate this functionality, we first import into the GPG keyring the private key previously used to encode some data. We would never include the private key in an Rmarkdown document in a real use case, but this is practical for this vignette and it demonstrates another use of text data chunks.

```{data key,output.file="key",include=FALSE}
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQVYBF6A3p8BDADcaf7tveXZUpi0IfEpmYrPP8/OSXSh3iBkd5bdTvbq/FwLGIsD
dp/dFqAWS+0BqCIMFAtV63FUOG4kXYpkajdl2QU1Hy0aY9F9K0imc5JUM1SEry5F
CckjzDFp3u4pmmCPWKF2jVnaHzahJfKz9J9qD9BfBSynfyQU2XgsrRqNgiqeNcOi
f0674hpReawnecBwhENKMWL38O1aOtP1IDx9cFI6busiiOaIHIYYW6qbv178offy
0OWogstsQ3EJQbPBPkkgVTn8wwGUtoorc/2AonSoz99QC4nMWbBaDUGuE9O32yRv
Q7Pe6bWVBuIeV5ASAfSSEypzNHB576BF6MTy+lJvhfXI41Yu97geQJM0CplJ8xav
xAhIvrKjkDoW3zwrZlG54G2TidwEyXoDx7cyRVnCf9tsBCmhEDiKvzlg2IE9Fo65
+LWrD12qCKi7cu4XE28q4zy7S4adhUCBcuflZ8wKMVvbZRXvqnAHBAK8gQxMqHMc
EjWAb7rvmN9bkTUAEQEAAQAL/if4vPeGYaGIvhKkuSRvKOIu01O4tIMKUluF6IEX
6eVxgIuulr85CwLAMKX6fO+4+vuvwuKBARth5G+J2ygcrxE0SyJ4FejcQ0hsyg8N
lHLaoDAzyLNSc/ye8jMd75jx2yMD0rw6JBpPYMvWou4JpcNJPOOOf6ucfgGd8pI/
jjotaecpHuJgLfoapeUyqIq8JK8C/WT+EdGfCpw7YObqQq4I6ZCZPuETbKMwcQ0H
yqfWC7bK9Lk/MvbdSWDH1j70f/t1KaUEBZ2z5xTALqxaFgbwXh+7FybzV+09Sxsn
l5deeubEQXwkbPthapjRpvRo197tJRHLJ8wQVCwag39ip5cvuWQIsej3qILKTepz
VBdgZa4hIyLX8uUCAtLrVYwvWzV1oWxPLAkXJ6KPCzB0jQb7q7UUyrBOUaavdnt2
aWBz5EuXPTaMqnzWqEKIazcXqiCSNjIEv7HWcU734IGUazYper3poYgOWYYIdUes
+xbdWP/j6313N3u4a9BSd3PMvQYA4CLwr+gBfX+dybX3jq3ldB3HJS/Lv90e64rh
BarRu+ByyEO5BcVJZ+ZEUOcBjF/pvG1qI9mfqBuZX/e2aW1lmMsxcXNlWRu5b5vE
geoRwqPMNIo4JIo2hByHZeEPQLcYW/QRy5xkoNbl+udPuS3PMEUnfnPeQKursY71
ao7Zo0TUeFRemEgkvxZpFXfT+IMs9DGI/Wi6PO0ChSJ/Cu/QixgK0eJFUroNCyvl
bW+xy0GSB325wkyOM5xIny681KtvBgD7v5V6n0P2UucxZYU5hhdWaaTf5aF83vtE
o88gSU5NRO1/wPFb+AFP3fw8TNtrvRlA/OakwjL+GbfhioAJ4mtPbdGUojFIAU6X
czMHbaYyNwZTMImBW9uc2gDqta8O1HiSwC7fXnTxVoSz3E/TD6dbAnFyf1FYNntJ
PLKS9H82idCqO0nrU3LtdKJx9VHJ6wLOT16D6zZAdgNB0wK9dzStayfIqQzN/FAz
01u0ehX4SDRCxxgukdR4ZyeZJfdmC5sF+wZ/2mW4Tp7v3kutNAytk4JtMvLIhe2r
BQkYw5eUFMq7tUqXgsXMjA0pVplUSosZknCIpoyoEU7rvS9BF9xdcpRixU5kxeYY
knQg5jtb+vx3Stpp0vbuvFFaGgEJhNP6Tg3al7gBCOwEEAJmSTko4cyf1e45pIMF
+jGbIeozSjeKPWjdJCr4q05tvKgsiAe7BulgUlNhS6Ty5JyQHsiM/WZTPko2BsN2
8Apa/nuOvYwRwFLGGXVVWV3jQroPI9Hbft9ctBhUZXN0IEtleSA8dGVzdEB0ZXN0
Lm9yZz6JAdQEEwEKAD4WIQTvl6O3A7Tx/z8NeR/qzEhnRW4g7QUCXoDenwIbAwUJ
A8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRDqzEhnRW4g7WxrC/94WT6J
HEEgyb9Bskm2ik+c/qUW8w7JgizYRi6jqi8+qiIesh99MZ/XPm5mgMTIvKr0z/IG
xaU+RKYFF5DqsAc4obg/ZmClOSY9FgDWlMEm7hEqourQxfJZXGWRNcU6DTr2tC/K
GpTNkhR802LnjUePeVJU5MMuJ8eyQV+NgGhwXTIcPA6ERwHIC1n24N3QDFNoijcc
pTi5p9+N33w8fBC5ZMeZwrWI6mCJjEWVbxG2zcsIJ2t7htWRM7W1rKi5lHRpQdn/
cd9WtbdDFj7ywGPnjMB2vxYVJreENGbE/LZIZPaJKJHPReWQ+GBSGkyY7nrT32SP
R+qj5gO0Bez7F+61EDU+SXP9PJ8fyTGtUWfTsgz+fTj2TDn39y0tL1wuSciEOAjD
uia+L5qiKE9GK6mBQv78yfzZ/ZOEdJn9ZNRWs8kvs/aG9BygYMdJM5T4vvk2DcWd
m061EGTg/AVUFpMuTon9tb+RCIFfVjSzat8LWcf4Me2nJeFZu+lW/lCmxkedBVgE
XoDenwEMANPff6PrZirginP4HNK7g3ANmB3bDKCI1msAQspXMzvhtMc0Hn8DpM+r
wPUuoOo4hnYwkGHSNZ4dulrtW99mlzQWcFwDuOsvPAqc/OuEIEo0BBvc5HcpNk4d
z94Vno+Dq904VnlStf6DXpGbBFZkZBoC4XVwFUSoEjD1i967ckjFUhOxE5ynlcMb
8mpS65iml4JFd572bcuo9exJ1g7IhdgFIFoDDD2eJkxEhmEHNiVd8B9/j1GHxDCq
v/D0HNbgKuFk8WJUMYvupdqA30wAc5Ujnf+nURfNejgZTOiGXm5FZBrw/dha7yTP
/mlnNFMBKUEBrxYyPo2JVSsYfPf1WzLL1dmv8JPC5fyEKYhEC+zBvlytRWqkZV88
DumgVEdhEnnMEVlofyF8KoVMmWYA9w/FUUKiNymZlK1PEGecqliEhXh+KE03ncHh
AyEo0Zcdh5sSxUW5fNsQb+tp0fqFBs7Yye432w6ID3ZIONrnWrQ6MewWwxeAGMam
x03jgyMlCwARAQABAAv9EJ0e8iicS1JuKOfUwsWHafr26ahqlhAE2EEd+6XY06JA
PbqdhZIwk0RBjjhIz/T8vjnSqIkGQU7NdSHVqW/u/VuhFeYI0xBSIfbrckBbE9Z+
V/z7QUjPBFMcIKsLUu+dQ2yOg1b0BHAis0I3ldqrasq9CStvz4FqY8JtZFrIfGJU
rEyfYBJYEQOY/7Ne3Ap8KO/vkFx8gZLPLecgTOp2bFkCj2xbwl0rXaGl8+fP3CBA
mweyok8GGFbbVDagKE1NiukpEVzHsoMyMfPkxdIMLSj0F2GzQSnhyhyGomNstuTT
EC/i3/u7M9TRvLkpNTP3I6z5VNjayrp0NBs0z3sb1wNzrACELWbTtb/Lo5BVVD9Y
m0MQtDi8+SKzTHci2AdpvewxnhO4IiS/aXYYGcPwmEX4YdlZeV0J5mRXNsvWxYZk
HHFkbfgUkiFSFOmb9uyPD0NMldJoLXbv9+LFiU1okglietVcKK7Fyt5xCKcxbtO8
kdYJTuWonsWeyC8tz1WBBgDcq6doxs3aFSVeLcZ0//WHif+iBYlLFoexmw4irx8e
LnZilDJ5i4mwcu6Q5qxao3UEyeUC7ff//Qn846TQMDDRcC3xtrbqAqVyYBE7u9EI
OMyyCfosk8nNmVBpNdnsFm76lUyG8GiuT6b0j8BiQTRPmH4Xlh3pSiihyuTJIVhX
Y663wV8EwT9IRnYCoVqw9s5qZqJGkI4rxnABuyJui4BpmkrLry70t1xb6MdX2BPD
eK5u0YJ24AmxPW5YGvXnO0sGAPXLRfarrI9IgSz28+QpfYttOIbjp3n3AxB3ImHo
oK+CLsc1vHtsdEV8hElWo9k5EqcdlhPBbeC6IILFqT69Ldx8jK85hxR0bYs2NVLC
qyWo1T3bovPePCEenN4++VPBtVBkEt51MByNIKwC3Bw0zvHcygLcHE3iXRQ40dhq
AZWrPlOqwnC8x9+UqZoWCp/JRWD5qBjD6EPVAxwbtcUdjDOhZ1y51xbUaX59Vlul
BGLse/0Q47m71HrF+d9rGUnlQQYAkDQsdbzijmB/tVzcRXJWbZVgjwLciofxVpoM
TEYyw8+oSYDI1L3Dikejp3XymVr+9pKGmPZjLqL9Q01J9epeHt5wgLjuWTXtkVLW
kbnt7vTy257BIsHGDwiJzMI7PujTlQ4B1ZTPz2WyUJ7gn1f+J9wYpNOr7qeE2pg6
cOeiPQmT5h88jWTUH/eAJ0nAWx46kwgQY4uZz7xsFtCcwQgqVe9bD5MNv/bBUdPW
RkF8ZbRCPRk4Vl2DYM/rXC2VGCFZ6OeJAbYEGAEKACAWIQTvl6O3A7Tx/z8NeR/q
zEhnRW4g7QUCXoDenwIbDAAKCRDqzEhnRW4g7ZayC/954y+kfmjtIzSRDBRpOo2s
npOOwy7RLdOdWvab6jVecyqYsDyd/fiCXVKxALOVR31WTef00iFSLHQactwFxQyJ
zY6YO8tGkvYEXXYJR5O5MNzjlhNMndBqGIbKe9tA2BFLDD/6mmvMD/i9k+IhHzFT
NhoczB5rE9oaApMZhAj9u9Uv2zy0osfcOPcy+RN9b2noodVS/7Ei2BjWl+V/MGqa
I8oBM/ETIW/jcq+OuE8oSqoByFtFHh1DgOzOFugCWApOmAjLQwQCmDiYYtKN1GWq
l1E+txLud78ZBsJQL/78MXO9V2T2dCbcIA0vOfACuoPApfu6seRE0SLeImgoRg+8
7aX6HtiRXRjExDS26YNbGYzAvVTl3Zy1VptXOMwkh5CcIgtTcDv32pLWC3xvNydG
P4xDMM+BVuDi6QTcFfbPtqYbuuT4OFyyaSzee0oWxvKoX2pL81VnMwvb7Uy47Dxf
Ng9Af4cf3nf9UzesAVbSy1gtvlZIyX0HwtZNVLNJSS4=
=C6UF
-----END PGP PRIVATE KEY BLOCK-----
```

This key then needs to be imported into the keyring:

gpg::gpg_import("key")

Now that we have this private key, we can decode some binary data encrypted using this key:

```{data t3,format="binary",encoding="gpg",output.var="d",loader.function=readRDS}
-----BEGIN PGP MESSAGE-----
Version: GnuPG v2

hQGMA9TPonHna5j3AQv7BIPNOSR/024iE0Gj3DCo3DvLvj/oEJ29XORHBkn4nul1
+zaRV5E/K4LCKxkkAEx/+FdM72x1hV5FF5Vf0FSet1RHiOOXPuChEEzRHOubkh/U
gw44Q72d8Dp6TOJ+1KT5k/fdkVKsOZRSttL8hvxqC4nyfObF0CkIoG+Kfx+kkYqu
araVWqNtcb3/FbtT+ZC0Hip0Ws6IJ8mGOhZdRxZ3S8KUtgf/t7S3Wa75c6L1wolT
R1/WhPgcWB4epLTvHdSmv9qcu/vFXE8SmNE5MV4V2aSTRU7y9WdPW/+XzU2Et4BK
kGyzhkI6q7QzAXFOeD1sn0uaUeH9/BDwn3AJZEXkwN4qaarPpDKjZ9GVE9Gg8521
BYe7AIZwq7sfnF+v1WyxamFYpSSAiNHze00MHPWot3Db+4SpRFYIWlYlZF00HIMo
Qspb4AmIfnNo9zj0RGG7GJoyod8ZrW4RF5iOEUWtyQ5z6LzymGTSdArWOq1fDgYW
tvEgbkbdYsJA6usJ3Zxc0sBJARfA6gDCFF72nGiAoNS98zoFjtD7hznY9DBvCOoF
jzJ8kHfPQPK9/bRVuofUDP+jOoJyf8/7eB6kANNq+XrzoZL0N42zHR2n47xupqYN
GSdsRljTra8a9zWIs9k8E5/79qRvV25c/wPeysulWkzhLCDaCMMVYJvQQ0JgT2L+
/eBtDeAuqkCqiDjeEGGB0Q4Q81IOIHMxUXFPJHvWJE4eJhnRjLmuPCBDQdL0JYTS
Bc48A/eA/cbfUr+4RluY9RLcaUHRPjKT8e7X98VdnSBPGvikVpSjR3zZhPNQs7Vb
C7H5lml4B8FpRgQBFwt2ou8URLRYR82tUa/OcsByW9jxf988YZx57a1hAw==
=m4sT
-----END PGP MESSAGE-----
```

d

Helper functions for creating and inserting data chunks

For small data chunks, one can copy-paste the (encoded) data from the command line or from a file into a Rmarkdown document. For larger data chunks, this can be awkward, so knitrdata includes two command-line helper functions for creating and inserting data chunks into Rmarkdown documents, as well as 3 RStudio Addins that facilitate creating data chunks. The command-line functions are create_chunk and insert_chunk. We can combine these functions with data_encode to generate the base64 data chunk described above. First, we use create_chunk to generate the chunk:

library(magrittr) # For pipe operator

chunk = data_encode("test.RDS","base64") %>%
  create_chunk(chunk_label="mydata",output.var="d",format="binary",echo=FALSE,
               loader.function=readRDS)

cat(chunk,sep="\n")
#> ```{data mydata, output.var = "d", format = "binary", echo = FALSE, loader.function = readRDS}
#> H4sIAAAAAAAAA4vgYmBgYGZgAWJmViCTgTU0xE3XAigmDOQwAfE7KM0IVMQJpPmS
#> 83MLEpNL4jPzSopTC9FkWZISi1OhYrxgcQj9D6QTZJUDBwMY2H9ApaFqBKDOAZnF
#> mAhjJMEYyUAbkK1jzUvMTS2G6mPCog9NeXJOYjFMOUyQKyWxJFEvrQhoEppyzqL8
#> cj2YDSBfMDUAif////8FuRYAeDNrwDsBAAA=
#> ```

Note that with the exception to the chunk label, chunk contents and other named arguments to this function, all the other arguments are not evaluated, so they can be given exactly as they should appear in the Rmarkdown document regardless of whether the chunk options make sense in the current context.

Next, we can use insert_chunk to place this chunk at a given line number in a Rmarkdown file (here taken to be example.Rmd):

rmd = insert_chunk(chunk,11,rmd.file="example.Rmd")
writeLines(rmd,"example_with_data_chunk.Rmd")

This will insert the new data chunk at line 11 in the Rmarkdown document.

knitrdata also includes the list_rmd_chunks and splice_rmd_by_chunk functions for identifying and potentially removing or working with the chunks in a Rmarkdown document. See the documentation and examples for these functions for more details on their use.

Data integrity checks using md5sum

Using the procedures above should assure that data inside data chunks do not have errors. Nevertheless, there is always the possibility that a stray keystroke would modify the data, particularly for binary data encoded as text. To protect against this, data chunks can have a md5sum chunk option that will be used to test if the decoded data matches expectations. A MD5 sum is essentially a very large number (typically encoded in hexadecimal) derived from a file’s contents that has a vanishingly small probability of being equal to the equivalent number derived from a different file’s contents. If two files have the same MD5 sum, they are almost certainly identical.

To use the md5sum chunk option, one must first determine the MD5 sum of the decoded source data:

tools::md5sum("test.RDS")
#>                           test.RDS 
#> "c5c4292e64e0536e69fd24862128e82d"

If this character string is given as the md5sum chunk option to a data chunk, then the MD5 sum of the decoded data of that chunk will be calculated and checked against this character string. If the two do not match, an error will be generated.

We can add a MD5 sum check to the data chunk generated in the previous section as follows:

md5 = tools::md5sum("test.RDS")
chunk = data_encode("test.RDS","base64") %>%
  create_chunk(chunk_label="mydata",output.var="d",format="binary",
               echo=FALSE,loader.function=readRDS,
               chunk_options_string = paste0("md5sum='",md5,"'"))

cat(chunk,sep="\n")
#> ```{data mydata, output.var = "d", format = "binary", echo = FALSE, loader.function = readRDS, md5sum='c5c4292e64e0536e69fd24862128e82d'}
#> H4sIAAAAAAAAA4vgYmBgYGZgAWJmViCTgTU0xE3XAigmDOQwAfE7KM0IVMQJpPmS
#> 83MLEpNL4jPzSopTC9FkWZISi1OhYrxgcQj9D6QTZJUDBwMY2H9ApaFqBKDOAZnF
#> mAhjJMEYyUAbkK1jzUvMTS2G6mPCog9NeXJOYjFMOUyQKyWxJFEvrQhoEppyzqL8
#> cj2YDSBfMDUAif////8FuRYAeDNrwDsBAAA=
#> ```

md5sum can be used on all data chunk types, including text data chunks that are not encoded in any special way.

Reading a data chunk from an external file

One disadvantage of using data chunks is that it can make Rmarkdown files long and difficult to navigate if you use lots of data. Rstudio can help with this problem by allowing one to hide the contents of a chunk and by using navigation tools to jump between sections. In addition, using insert_chunk or the RStudio Addins to place a data chunk inside a document as the last step before public diffusion may avoid having to work extensively with the large file. Nevertheless, large file size can still be a problem. To facilitate the initial construction of standalone Rmarkdown documents, data chunks can be read from external files using the external.file chunk option. The external file must contain the encoded chunk contents exactly as they would appear in a data chunk. The intended use of this option is that large data chunks would be placed initially in external files, but the contents of these files would be placed directly in the data chunks before sharing the document with others.

writeLines(c("This is from an external file.","It has two lines."),
           "test_external.txt")

```{data ext,output.var="ext",external.file="test_external.txt"}
Content will be ignored with a warning!
```

cat(ext)
#> This is from an external file.
#> It has two lines.

Extra language engines for CSV and RDS data

As it is very common for data chunks to be used for CSV or RDS data, additional shorthand language engines are included in the knitrdata package for working with these data. These chunks types are csv, csv2 and rds for working with comma-separated CSV data, semicolon-separated CSV data and RDS data, respectively. These language engines are implemented as wrappers for the standard data language engine, but with the loader.function option predefined to be read.csv, read.csv2 and readRDS, respectively. Extra parameters for these loader functions can be given directly as chunk options, as well as via the loader.ops chunk option.

As an example, one could load CSV data that has no header and that uses a non-standard delimiter and comma as the decimal point indicator as follows:

```{csv2 output.var = "d", sep="|", header=FALSE}
a|1,1
b|2,2
c|3,3
```

d
#>   V1  V2
#> 1  a 1.1
#> 2  b 2.2
#> 3  c 3.3

Numeric vector data

You can use a data chunk to load numeric vector data as follows:

```{data output.var="v",line.sep=""}
1.2,3.4,5.6,
7.8,9.0
```

```{r}
f = function(v) as.numeric(strsplit(v,",")[[1]])
v=f(v)
v
```

#> [1] 1.2 3.4 5.6 7.8 9.0

Conditional text in Rmarkdown with missing values

One way to create large amounts of text in an Rmarkdown document that is only included in the document if certain conditions are met is with the asis language engine. For example, if one creates an input parameter in the YAML header of an Rmarkdown document:

params:
  cond: TRUE

Then one could include content in the document based on this parameter as follows:

```{asis eval=params$cond}
# A conditional section

This **will not** be evaluated: $5+4=9$
```

This will add a section and a paragraph with one sentence. One disadvantage of this approach is that the content of an asis chunk is never evaluated so one cannot use inline R code.

This limitation can be overcome with knitrdata by pushing the content of the chunk into a text variable with conversion specifications that can be replaced using the sprintf function. For example, placing the following in the document:

```{data include=FALSE,output.var="cond_text"}
# A conditional section

This **will** be evaluated: $5+4=%d$
```

```{r results='asis',echo=FALSE,eval=params$cond}
cat(sprintf(cond_text,5+4))
```

Will lead to %d being replaced with 9.

Unencoded text chunks and md5sum

Using md5sum checks on text chunks that are not encoded (i.e., 'asis') can be tricky because when the chunk is processed, an operating-system-dependent newline character will be added to each line, including the final line, and character encoding (i.e., UTF-8, latin1, etc.) translation might be carried out. This will potentially make the md5sum check fail if the file was created on a different operating system or the final line lacks a newline character. To have the best possible chances of success when using md5sum checks with unencoded text files, follow these steps:

• Assure that the original external text file has a newline character on the last line (i.e., the file ends on a new empty line).
• If you do not care about cross-platform compatibility, then just make sure that the original external text file used to calculate the md5sum is formatted as is standard for your operating system (i.e., newlines with '\r\n' on DOS/Windows, and newlines with '\n' on all other operating systems).
• For cross-platform compatibility, specify the line.sep chunk option to explicitly set the newline format to use. For example, you can specify chunk option line.sep="\r\n" to force the data language engine to output DOS formatted text files.
• Stick to a single character encoding for all text files and the operating systems being used, ideally UTF-8.

If you follow these steps, md5sum checks of text chunk contents should work. The Insert filled data chunk RStudio addin will attempt to set the line.sep chunk option to match that of the incoming text file so that one normally should not need to worry about the text file format.

If for some reason these steps are impractical or are not sufficient (e.g., due to differences in character encodings between operating systems), one can always use a different method to check the validity of chunk contents. Base64 encoding of text files with a md5sum check in a data chunk is always a possibility, but if content visibility is important, then one can use other hash algorithms on the output of a data chunk. For example, the following would do a post hoc check of the validity of loaded data using a SHA1 hash:

```{data output.var="d",loader.function=read.csv}
a,b
1,a
2,b
3,c
```

```{r checkdata}
if (digest::sha1(d) != "a5918a84b39f0b6f42e9ab4a19771a7d6a5777a0")
  stop("data corrupted!")
```

md5sum checks on base64 or gpg encoded text files are generally reliable even if the above steps are not followed as the exact text file contents are encoded in the chunk.

Workarounds for GPG data chunk error: Password callback did not return a string value

If one includes in a Rmarkdown document a GPG-encoded data chunk that uses a password-protected GPG key, then knitting may fail with the error Password callback did not return a string value. This is because knitting of an Rmarkdown document takes place in a non-interactive R session and, therefore, gpg::gpg_decrypt (via knitrdata::data_decode) is unable to open the password entry dialog during the knitting process. The gpg package does not currently include another mechanism for providing the key password, but one can work around this problem by using any mechanism to temporarily store the key password in the GPG keyring manager. This can generally be achieved by decrypting something that was encrypted by the same key that was used to encrypt the chunk. This normally leads to the GPG key password being temporarily stored in the keyring so that gpg::gpg_decrypt will not attempt to open the password entry dialog, thereby avoiding the problem.

As a workaround for this problem, knitrdata includes the unlock_gpg_key_passphrase that when run from the command line with identifying information regarding the key needed to decrypt the data chunks in the Rmarkdown document will attempt to unlock the key.

knitrdata::unlock_gpg_key_passphrase(name="David M. Kaplan")

It achieves this by encrypting and then immediately decrypting a small amount of data with the given key.