InMemoryDatasets.jl

Welcome to the InMemoryDatasets.jl documentation!

This resource aims to teach you everything you need to know to get up and running with the InMemoryDatasets.jl package.

InMemoryDatasets is a collection of tools for working (manipulating, wrangling, cleaning, summarising,...) with tabular data in Julia.

If you are new to InMemoryDatasets.jl, probably First steps with Datasets or Tutorial in manual should be good starting points.

Package manual

• First steps with Datasets
• • Setting up the environment
  • Creating a data set
  • Basic utility functions
• Tutorial
• • Getting and loading the data
  • Filtering the data
  • Select: pick columns by name
  • Applying several operations
  • Reorder rows
  • Add new columns
  • Applying functions on each group of observations
  • Non-reduction functions
  • Visualising your data
  • Exporting data
• How InMemoryDatasets treats missing values?
• • Comparing data sets
  • Every column supports missing
  • Functions which skip missing values
  • Multithreaded functions
  • Other functions
• Formats
• • Introduction
  • Manipulating formats
  • Modifying a data set
  • Using Dictionary
  • format validation
• Function map/map!
• • Introduction
  • map!
  • map
• Function byrow
• • Introduction
  • Optimised operations
  • mapreduce
  • User defined operations
  • Special operations
• Transforming data sets
• • Introduction
  • Specifying the transformation
  • Accessing to modified columns
  • Specialised functions
  • Using Tuple of column names
• Filter observations
• • Introduction
  • byrow
  • mask
  • Using contains and related functions
  • Julia broadcasting
  • Related functions
• Sorting Datasets
• • Introduction
  • sort!/sort
  • sortperm
  • unsort!
  • issorted/issorted!
  • Performance improvement using formats
• Group observations
• • Introduction
  • groupby! and groupby
  • ungroup!
  • gatherby
  • Iterate eachgroup
• Aggregation
• • Introduction
  • combine
• Reshape data sets
• • Introduction
  • Simple case
  • transpose of grouped data sets
  • Advanced options
  • Spreadsheet-style pivot table
• Joins and related topics
• • Introduction
  • Database like joins
  • contains
  • Close match join
  • Inequality-kind joins
  • Update a data set by values from another data set
  • compare
• Gallery
• • General
  • map! and map
  • Row operations, byrow
  • Filtering
  • Grouping
  • Joins
  • Reshape
  • for loops
• Performance tips
• • Make use of provided APIs
  • Avoid pushing single values
  • Avoid using String type for large data sets
  • Choose groupby or gatherby based on the problem in hand
  • Beware that every column must support missings
  • Master byrow
  • Don't avoid for loops!

API

Only exported (i.e. available for use without InMemoryDatasets. qualifier after loading the InMemoryDatasets.jl package with using InMemoryDatasets) types and functions are considered a part of the public API of the InMemoryDatasets.jl package. In general all such objects are documented in this manual (in case some documentation is missing please kindly report an issue here).

Please be warned that while Julia allows you to access internal functions or types of InMemoryDatasets.jl these can change without warning between versions of InMemoryDatasets.jl. In particular it is not safe to directly access fields of types that are a part of public API of the InMemoryDatasets.jl package using e.g. the getfield function. Whenever some operation on fields of defined types is considered allowed an appropriate exported function should be used instead.