I do my development work and run most of my agency (with multiple employees) with Linux. For some of the business things that most people don’t assume Linux can do, I use:
1. Xournal to annotate PDFs (aka sign contracts without printing them and scanning them back).
2. LibreOffice of course for most document related work
3. OBS Studio for recording webcam videos along with screen sharing
4. Audacity for audio editing (heck, I used this even when I was on Mac OS X)
5. Technically I’ve tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now
6. pdfsandwich and Tesseract OCR for OCR/turning PDFs into searchable files
7. Chrome/Firefox are both first class and run all the modern day web application stuff
8. Tons and tons and tons of command line stuff that Linux is well known for
9. QEMU/KVM for hosting arbitrary virtual machines with almost native performance
10. GnuCash for double entry accounting for personal and volunteer society finances. I used GnuCash for a while to run the S Corp accounting when we were on Freshbooks and Harvest, but we’ve since graduated to QuickBooks Online for better invoicing and CPA office professional services support.
11. GIMP for photo editing
12. Inkscape for messing around with vector graphics
Once you get past locating the tools to do your job. Linux has everything.
1. “Xournal” is amazing, small install footprint, simple interface and can do everything you need with pdf’s.
5. You should try [1] “Kdenlive”, it has it’s quirks (as all linux tools do) but it’s probably the best open-source Linux video editor out there.
[1] https://kdenlive.org/en/
For video editing, Blender is the best FOSS tool I’ve tried.
But you now get Davinci Resolve for free on Linux. It blows the FOSS competition out of the water, being a software with probably millions of dev hours funded by Hollywood studios behind it. Blackmagic decided to go the way of providing the basic tool for free in order to build user base amongst hobby video editors, and it’s not a bad move IMO.
Wow, thank you for that, I didn’t know Resolve. I do use KDEnlive but I can’t say I’m satisfied with it.
Resolve has a quite steep learning curve, at least it had for me without experience with other NLE. For my simple workflow it was too much and I backed-off.
Happy to see the recommendations here for video editors on Linux, will give some of them a try.
Sorry for piggy-backing on this thread, but is there a simple tool which will let me do a screen capture and annotate it. Something like ScreenFlow on Mac. I use Peek for screen-capture currently.
I wanted to annotate some text and OpenShot doesn’t really work well for that as I can’t place the text freely and am limited to a few templates.
I had too many headaches with Resolve 16.2 hanging on exports or UI stopping responding on Ubuntu 18.04, I hope it gets better. The camera stabilization filter is amazing.
Kdenlive does support MP4 as source assets, Resolve free only does ProRes MOVs and such. It’s just an extra step with ffmpeg but still.
A little out of topic, but which tool would you recommend for just the basic video editing on Linux? Nothing fancy, just cutting and pasting videos together, possibly with some simple transitions? Thanks!
My wife (a music teacher) was a Mac only video editing person. She now uses Open Shot almost exclusively, even though for $40 I bought her Sony Vega (she was complaining about Open Shot at the time). I think Open Shot is a good gateway to Blender or other video editing tools. She does all her stay at home videos on Open Shot now because she can do it so quickly. I would say it took about a week to get really comfortable making simple videos.
Not OP, but I think OpenShot is made exactly for this kind of simpler editing. It’s akin to Windows Movie Maker.
ffmpeg – you can do declarative editing combined with a Makefile. Clips can be previewed with `mpv –osd-fracitons` or equivalent.
Avaliability of Davinci on Linux is impressive. I hope other productivity products do the same thing.
I have tried many video editors including OpenShot, Cinelerra and Blender to finally arrive to Kdenlive and stay with it! It is still in an active development and getting better and better! I also definitely recommend trying anyone Kdenlive.
“Quirks” is an understatement. Last I checked, you couldn’t even have multiple timelines in the same project, every bit of editing had to be in the one same timeline.
This feels off topic. I agree with these suggestions and appreciate the suggestion for Xournal, will look into that.
But TFA focused on fundamental command line tools and shells, not office productivity apps.
There are thousands of articles on the classical command line tools grep, awk etc. With a subject “Linux Productivity Tools” I think this was both on topic and valuable.
+1 I find the discussion here more appropriate to the “Linux Productivity Tools” title than the original article.
1. Xournal to annotate PDFs (aka sign contracts without printing them and scanning them back).
Will check it out. For PDFs I mostly use Master PDF Editor
2. LibreOffice of course for most document related work
For me LibreOffice is a bad nightmare. I use the commercial Softmaker.
6. pdfsandwich and Tesseract OCR for OCR/turning PDFs into searchable files
Good luck with that. My results with Tesseract were always abysmal. I use ABBYY Finereader with wine. I would pay for a native Linux version. They have a Linux command line tool that has a biblical price tag.
12. Inkscape for messing around with vector graphics
Inkscape is good. I wish they would still develop Xara XL
What I am missing in your List:
Recoll. Find Stuff on your computer. One of my most important tools.
This is an excellent list. I just want to add that jetbrains products (intellij, webstorm, etc) all run perfectly well on Ubuntu (and probably other distros?), with regular releases using their toolbox app.
Can definitely vouch for Shotcut. I tried everything I could find a year or so ago, and Shotcut was by far the nicest to work with.
LaTeX is great for professional-looking typesetting, and it has the power to do just about any layout tricks you want, but, in practice, you’ll be fighting an uphill battle if you try to trick it into doing something that someone out there hasn’t already written a package for.
(I say this as a professional mathematician, who lives his life in LaTeX. It’s fantastic for writing math, and trusting that all the kerning etc. will be handled properly. However, when I want any formatting tricks, even after 25 years I still have to turn to my local TeX guru, who more often than not says “you don’t really want to do that with TeX.”)
Not sure what formatting tricks you mean, but that doesn’t seem quite fair. When I want to do anything unusual, I google my problem and almost always there’s a question on Tex stack exchange (which is blessed with the participation of most of the experts in the subject) with answers giving several easy ways to do it, using packages I already have on my computer! And it’s easy to make your own commands when existing ones don’t do the job.
The canonical one is forcing image placements. Sure you can Google and get a result, but all of them are prefixed with “if you _realy_ wany to do this here’s my preferred workaround, but you should let Tex do the type setting”.
> When I want to do anything unusual, I google my problem and almost always there’s a question on Tex stack exchange (which is blessed with the participation of most of the experts in the subject) with answers giving several easy ways to do it, using packages I already have on my computer!
Yes, exactly! As I said:
> if you try to trick it into doing something that someone out there hasn’t already written a package for.
There’s an incredible package library out there, rivalling CPAN, and I love TeX and won’t speak against it; but, if you try to step outside the package library (or even if you try to compose packages in sensible-seeming ways), as is very easy to do if you try to view TeX as a general-purpose typesetter, it rapidly becomes clear that there’s a lot of magic going on that those packages hide away (more or less neatly, depending on their maturity), and that is hard to reproduce on your own, or add to.
I would say the vast majority of business documents, proposals, contracts, etc are collaborative editing with others. I didn’t mention it but basically Google Docs and Sheets rules the world with these given we have a 50 user grandfathered account. For contracts, end result is exported to PDF and signed with Xournal and sent to the client for countersignature.
I meant to use LaTeX beamer to build these slides but couldn’t figure an easy way to make the “Anatomy” figures eg. on slides 8,14,15 etc.
Could try out Blender for your video editing.
It also appears that DaVinci Resolve has support for Linux.
Does anyone know of some modern blender tutorials aimed specifically at video editing?
if possible, including capturing footage form the screen (screen casts).
I’m showing my bias but my good friend Andrew runs Blender Guru on YouTube. Admittedly he covers everything from beginner to advanced so I don’t know where your skill level sits, but Andrew has a lot of content.
I had totally forgotten about Blender since I never really understood 3D animation. But great idea. Thanks for both suggestions.
Not affiliated in anyway. Just a very happy customer, but want to plug PDF Studio [1]. I’ve used Atril, Evince, Adobe, PDFXChange (through WINE) and this PDF Studio (not FOSS) is worth every penny. It even interfaces with docusign (but you don’t need it since you can import real digital signature) their support is amazing, and linux is first class there.
[1] https://www.qoppa.com/pdfstudio/
> 5. Technically I’ve tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now
I wish that there was a better video editing alternative for a non-mac… Openshot and the others I tried crash too much (Openshot crashed for me today while trying to do simple trim)
The only one that I could always make work is Kdenlive. That said I never tried DaVinci Resolve as it’s a bit beyond my use-cases, but I’ve heard good things about it.
Have you tried Davinci Resolve? It is one of the top 3 video editing suites, and it actually even runs on Linux native! It is even supported on redhat (but works on others too). Very stable as well (unlike Adobe’s jokes).
I agree with you regarding crashes. Kdenlive has been much more stable for me but I don’t really like it. Can’t say exactly why but it just feels clunky.
I’ve not had an issue with openshot personally, though I haven’t had to much more than trimming.
I use arch linux, so whatever the latest version released on that.
The new 2.5.1 version on OpenShot is a big stability improvement over 2.4 (which is the current repo version)
For 5, good alternatives: Shotcut and Kdenlive.
and 13. Krita for digital painting
I’ll also add Olive[1] as an alternative video editor. I tried shotcut this weekend for just a simple clip/split mashup of videos and it was so slow and cumbersome. Olive on the other hand was extremely responsive.
[1] https://www.olivevideoeditor.org/
Thanks for this list! Wish I knew about Xournal earlier this year.
Care to share any details about pdfsandwich and tesseract? As in, do you have some scripts to glue it all together?
Xournal++ is also worth a look. It’s a fork that includes a few features I missed in the original, otherwise it’s almost exactly the same.
For OCR I use OCRmyPDF[0] on my Mac (also available for Linux and Windows). It does a very good job, is pretty fast, can even OCR images by converting them to PDF and you can use different languages. It also reduces the file size significantly. I really like it.
[0]https://github.com/jbarlow83/OCRmyPDF
I’m a big fan of GNOME Boxes. The developers put a lot of care into the UI design and into choosing sensible defaults so most things work well out of the box. However, there are some kinds of advanced tasks that can only be done via virt-manager. I doesn’t happen very often for me but when it happens it can be useful to know how to use virt-manager to configure the VMs that GNOME Boxes created.
But you know what bothers me? Virt-manager is mature and feature-complete, and RedHat has discontinued it in favour of cockpit.
Cockpit is cool and I like it, but it’s not as complete as virt-manager.
Great list!
> 5. Technically I’ve tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now
Have you tried Kdenlive lately?
In certain forms, text need to be filled in a series of square boxes. If there is an option to adjust character spacing, you can in fact fill such fields too using a monsospace font. But I am yet to come across an open source (or even free) tool (for annotating PDFs) which can adjust character spacing too. Can Xournal++ do it? If not, are there any other open source or free tools that can manage it?
Yes, only that it has to be done page by page and then you have to merge the pages back.
> 8. Tons and tons and tons of command line stuff that Linux is well known for
And this is what covered by “Linux Productivity Tools” slides.
Kinda like this guy except I use google apps for documents and sheets, VSCode for an IDE and…
I boot Win10 for Steam games, Fusion360 and AutoCAD.
There’s so many games on Steam that run native on Steam or really well via Proton. I abandoned Windows a few years ago with video games being my last hold-out. If it doesn’t work under Proton or isn’t native, the game is a hard pass for me.
To each his own. Windows isn’t that awful these days and wrestling with Nvidia drivers on Linux is a PITA and entirely Nvidia’s fault.
Their drivers on Windows aren’t that great, either.
I’m in a very similar position and have had much the same positive experience you’ve had. I use most of the tools you listed plus a few others like Gitkraken (git gui), Blender (3d effects, logo, video overlays) and ffmpeg (occasional video optimization or editing).
Instead of Gitkraken I recommend GitG. Gitkraken is both sluggish and absolutely guzzles memory, which isn’t unsurprising considering its another Electron application.
It’s like suggesting Nuke to do video editing, while possible, not the easiest and most likely you need previous experience with the software.
Thank you for Xournal. I used LibreOffice for that but it added artifacts to my images (like a vertical black line to one if rendered as a PDF but not if rendered as a TIFF).
I’ve been using Flowblade for video editing lately, and have been very impressed (granted, my needs are pretty simple).
This sort of sounds like a vegan/vegetarian telling me that the veggie chicken tenders taste a lot like regular chicken tenders.
Maybe I should have just downvoted and left it at that, but it’s late at night, so here goes.
How exactly do you think a comment like this contributes to the discussion? Someone posted a list of tools they use. You reply with this. The world would be better off without this type of information-free negativity.
Twenty-five-year Linux user and lover here, and heck, I got a good chuckle out of the guy’s comment.
this analogy works on a deeper level than you think because you can make some really great vegan meat replacements although the majority of people just stubbornly refuses to believe we’re there yet
vegetarian meat (usualy from industrial satay, made with dissolved soy in whatever chemical bath) is what drives me away from pretty much every vegetarian/vegan place.
this is why opensource that is just a cargo cult to copying even the bad decisions of commercial software harms more than help.
id be fine tasting a nice indian meal made with vegetables. but instead I get gnome changing the side of the window close buttons (while at the same time removing the options dialog to change it back) just because the designer du jour liked copying osx instead of windows. it’s fake-meat all over again.
This makes no sense. Gnome has its buttons on the same side as Windows. You are probably thinking about ubuntu which patched gnome to move the buttons to the left.
And even ignoring that detail, the side the window buttons are on is entirely made up. Windows isn’t the real OS with OSX as the fake windows clone. The gnome philosophy is to support a minimal number of configurations but to make sure they are all tested and work perfect. Other DEs allow full customizability but I have found them to be buggy.
How many desktop environments actually let you switch the window button side? I haven’t seen one, and if you know one, why are you using gnome instead of it?
> How many desktop environments actually let you switch the window button side?
KDE does. And Gnome used to.
Give KDE a try. Gnome has steadily dumbed down and removed features; KDE is delightfully customizeable and very, very polished in current form.
This. After the KDE 4 debacle, I gave up on KDE because they cut all the functionality I relied on. However, I’ve been pleasantly surprised by how functional the newer KDE5/Plasma desktop is, especially once you change a handful of really ugly defaults (mouse cursors, window switcher and a couple other similar things)
That, imho, is the major boat anchor holding KDE down: the ugly defaults. If they would take a moment to apply tasteful default settings it would make a huge difference in the marketing value.
In the end, though, KDE is “just like Linux” in the philosophy of “you don’t like it? change it!”
Out of the box, though, KDE Neon or Magneia are nice enough. The new Breeze theme is much better than Oxygen. What concerns me a lot more is that major features like Activities don’t “just work” on major Linux distributions (e.g. on Debian testing, on my desktop, trying to create a new activity just sort of hangs and causes a daemon’s cpu usage to spike to 100%
Interesting conundrum, with KDE I’m fighting the abundance of features not to break things but with GNOME I’m fighting the absence of features in order to make it productive.
keep in mind that the tweaks app was a very voiced project against the gnome team. it kept fixing what they broke. it was mostly a fyou message …that everyone must use daily, which say a lot about the message.
I can’t remember the last time I clicked on a close/minimize/maximize button. May be when I was in middle school 20 years ago.
Its table of contents doesn’t do it justice. this is a great deck! super dense and straightforward.
my tiny tip to contribute:
for those who could never be bothered to remember the ctrl-commands for traversing words, but are familiar with `vi`-style movements: you can go one step above the suggestion in this deck with `set -o vi`.
I’m a heavy vim user, but I still prefer the emacs-style key bindings for bash. If I want to do vim-style editing on the command line, I sometimes use Ctrl-X Ctrl-E to edit the line in my default editor (which is vim).
YMMV, of course.
Same here, I’m more productive using directly the GNU readlines shortcuts, but it’s maybe a matter of habit
`.inputrc` is your friend. I have `v` (in normal mode) configured to open the line in an editor, because muscle memory sometimes makes me try to enter visual mode when editing a command.
fc is an interesting related bash-builtin command. It will open $EDITOR and put the last command in it to edit.
I do the same. This might be overboard to some, but I went a step further and installed a vim plugin that brought many of the emacs-style key bindings into vim.
Thanks for the praise. The slides are kind of a summary of everything I learned here at HN over the years. Feels great to be on the front page. Definitely going to brag about it for a few years :).
Another is C-x C-e to invoke $EDITOR (or $VISUAL, if defined), and go to twon in your preferred editor.
Like others, I use vi over emacs, but prefer emacs Readline bindings … to a point. Full true editor at the shell is quite handy.
> you can go one step above the suggestion in this deck with `set -o vi`
Highly recommended if you use Vim regularly (and you can define the same escape key(s) for your terminal and vim instance). For me that was one of the biggest productivity boosts when doing editing and command line work.
vi mode plus swapping my capslock and escape keys are by far the best thing I’ve ever done for terminal productivity. it kind of sucks on shared dev servers though.
the great thing about this is you dont even need vim, its built into bash.
locate is not mentioned — two orders of magnitude faster than find for searches from ‘/’ on my laptop, e.g.
find / -name *bashrclocate bashrc
Faster because reading from a single binary cache file, updated nightly, rather than opening every directory inode under /. To force an immediate synchronous update of the cache:
sudo updatedb
Relying on a database which gets updated nightly is problematic for some usage though in at least 2 ways: I don’t leave my computer on if I’m not using it so there’s a good chance the db will simply never update (I guess?) plus it’s only usable for locating files with confidence if you know they’re system files or else they were on the system yesterday. Or, you run updatedb manually all the time and weep because that’s not something instant, at all.
All in all, the idea is good, but I’m almost sure there are better implementations (even Windows has tools like ‘Everything’ which use a database but which gets continuously updated).
There are cron implementations that handle the “turned off at night” problem. anacron is one of them.
Even running updatedb (except the initial run) and then running locate might be faster than running a recursive grep from the root directory. Definitely if you run that grep more than once.
>you run updatedb manually all the time and weep because that’s not something instant, at all.
No, not instant, YMMV. On my mid-range laptop with SATA SSD:
$ time sudo updatedb
real 0m0.501s
user 0m0.247s
sys 0m0.218s
Is there any alternatives like “Everyting” on Windows? I really like it for its GUI and depending NTFS index which is immediately updated.
curious, is there a rust based implementation of posix utils or at least stuff from gnu-coreutils etc. ? thanks !
locate relies on indexing crons. This has killed several production servers until I disabled it.
Nice slides. I missed him covering ripgrep and friends, which is very useful to avoid slower find-grep queries.
Along with ripgrep, I think GNU Parallel (covered in the slides) and htop (briefly discussed) are great additions to any Unix development environment.
The rest of the standard utilities have stood the test of time surprisingly well. But top is a bit wonky, xargs has many pitfalls and, as I said, ripgrep is great for speeding up some find-grep workflows.
+1 for ripgrep (it really is a lot faster).
Also, I recommend using fd as a replacement for find. It’s API is a bit easier to grok imo, and it’s similarly blazing fast.
fd/rg/fzf are the core tools I end up using most frequently, working in a big monorepo.
Does fd actually cover all the same ground as find? I used to be intimidated by find, but these days I find it’s extremely useful for all sorts of otherwise complicated file location operations. For example, to find files newer than a certain date, just use ‘touch’ to set the date of a temp file and then ‘find -newer temp’. Or, I have a script that deduplicates all the regular files in a tree by hard linking them to a file named by the sha256 of the file’s contents ‘find -type f | ( while read -r fn; do …; done )’ and then, after deleting files or whatever, I can “garbage collect” the shas by deleting files in my link farm that only have one hard link ‘find -links 1 -print -delete’
Much the same here, but I tried fd the other week, and for simple searches it definitely feels much quicker and “lighter”, so worth adding to the toolkit imo.
Here’s my non-cli list of to me usefuly software on linux (working as freelancer since 2010):
– Webstorm + Phpstorm: Amazing tools for web development
– Sublime text for simple text editing
– Meld for graphical diffs between files and directories. Very useful.
– Signal client: Using it more and more recently
– Spotify: That’s not productivity though 🙂
– Joplin for markdown formatted note taking
– KeepassCX + FF extension: Works decently for password management. Though there’s no solution for FF mobile as far as I know.
– Reaper for professional audio production (never liked audacity and ardour). There’s also Bitwig which is more like Ableton live but quite expensive.
– Gimp: I use it but I don’t like the user experience.
– Inkscape works pretty well for anything vector based
– Libreoffice: I tend to use calc regularly
– Darktable for raw image editing
– Arduino
– Figma for digital design (web-based, so not really linux)
– Nextcloud as file storage and for synching contacts and calendars.
– Zoom for video conferencing (need to try jitsi again though)
> KeepassCX + FF extension: Works decently for password management. Though there’s no solution for FF mobile as far as I know.
Not sure what mobile platform you use, but on Android I have been very happy with Keepass2Android (https://github.com/PhilippC/keepass2android). It uses the android password-manager API so you get autofill (or at least a quick-access button) on login fields in both apps and websites.
Edit: It also supports a lot of methods for syncing the keepass database between devices, I point it to a directory that’s managed by Syncthing and it works pretty flawlessly.
I’ve tried Joplin but have stuck with standard notes for now.
Joplin is very good and has basic things like ctrl+f for finding withing a document that Standard Notes lack, but end to end encryption is baked in to SN where as it’s somewhat trickier to set up in Joplin. Both are solid recommendations to check out though.
One useful thing when managing one’s time is to know when a command will complete. I have a habit of using “pv” anywhere I wait for a result so that I know how long it takes:
http://www.ivarch.com/programs/pv.shtml
It’s basically “cat”, but with a progress bar. Also useful for measuring IO speed and volume (e.g. decompression: `pv -c largefile | gzip | pv -c> /dev/null` )
You can also use it with dd, or even just redirect it directly to a block device. I use it a lot for writing USB images. `pv image.iso> /dev/sdX`. Just gotta be careful not to redirect it to the wrong one…
Project idea: ML-based snafu-avoider that prompts for confirmation when past invocations by other users in the community of that command resulted in manifestations of regret[1].
[1]: screaming picked up by mic; detected maniacal CTRL-C’ing as immediate as it is futile; permanent loss of peer relationship to the project’s cloud (indicating subsequent reformatting and/or booting-off-tall-building)
Latest versions of dd support the argument “status=progress” for displaying the average write speed while running.
I use bmaptool for that. It shows progress and prevents writing to mounted volumes.
This is an awesome deck. I’ll be bookmarking this as a great resource to pass along.
Tangent: although the portability and consistency of PDFs is nice, I’d love a browser extension that gave me a nicer view of slides that were exported as a PDF. I find there’s something more satisfying about discrete pagination vs continuous scroll for these types of things.
On Firefox and maybe on chrome, if you set the page height to fit, it will go in presentation mode and space bar goes to the next slide
Are there any good parsers to convert pdf to html/epub? It would rely on the pdf being well structured.
Don’t most PDF viewers? Apple’s native PDF viewer (which isn’t available on Linux, so not a solution here, but which isn’t terribly feature-rich so that I assume that whatever it can do most any browser can do) switches at a keypress among continuous-scroll, one-page, and two-page views.
I like jq (https://stedolan.github.io/jq/) for this purpose.
It’s a very flexible tool for filtering, querying and manipulating JSON data, but when run with no arguments, it performs a no-op transformation that just pretty-prints its input.
jq doesn’t come pre-installed with every OS, python’s json.tool does. So it’s always there when you need it, no installation required, which is why I put it the same category of all those classic command-line tools.
I don’t actually believe my last Linux install came with a system Python. And as a Python developer I always end up using some tool or another to manage installations of multiple Python versions anyway. Furthermore jq just works as a classic command-line tool.
> … just works as a classic command-line tool.
impossible to understate the value in: not frequently fucking up your ability to run the critical/mundane/everyday/musclememory commands you’ve come to rely on after maybe 100+ invocations/week as far back as you can remember just because someone you’ve never met wanted to bump the .ruby-version in your java monorepo
What? Half of Ubuntu’s internal stuff is written in Python. They’ve switched to Python 3 but I’ve never heard of Ubuntu without Python.
I’m sure that’s true and maybe I’m misremembering but I still think I had to apt install a Python runtime I could actually invoke from the shell.
jq is yet another bloated syntax to learn and I’d definitely refrain from suggesting it to anyone whos not working with json everyday.
I tend to agree, regretfully.
I’ve unfortunately often seen coworkers spend half an hour painstakingly crafting and debugging a jq query, that could have been cobbled together in a few minutes with grep and sed.
I think the issue is that jq’s interface makes it really hard to “narrow in” on the right solution, whereas grep is much more forgiving.
I agree in the general case, but the syntax for pretty printing which is what he is doing is literally nothing.
Or ‘jq’, if you include the binary name.
Hardly “bloated”.
i dont know if i’d call its syntax bloated at all.
jq is something i’ve hated learning but powerful beyond tying grep and awk together.
jq is a tool, and like every tool, requires some upfront learning. Once mastered its much more productive than the grep/awk/friends solution.
for instance, to grab the rgb field for the color green…
jq -r ‘.items | .[] | select(.name==”green”) | .rgb
An alternative would be to use XPath 3.1.
I am implementing XPath in a tool called Xidel.
There the syntax for the example is much simpler: xidel – -e ‘?items ?[?name=”green”] ?rgb’
Depends what kabacha means with bloated. The XPath syntax is more verbose, but less cryptic
Agreed about the bloated syntax bit. Whenever I need to work with jq , I tend to trial with jid first and then use the actual syntax with jq. Convoluted if you think about it….
Invocation from command line:
$ firefox --new-window somefile.json
The viewer does however suffer from poor scaling with file size.
but you are out of command-line once you do that. The whole point of this guide is that you can combine all these tools together.
Why go to all that extra effort to type a “ $ python3 -m json.tool test.json
Bit of pointless contrarian opinion from a long time cat|’r:
> Characters saved with ” Hours lost with accidental “>”? unbounded.
Preface: I’m confident on the shell & with my keyboard: I’ve been using shell redirects for a couple decades, learned touch-typing up to around 80WPM, and used to note-take math lectures live in latex. Yes, I have noclobber enabled. But…
I can’t promise myself I’m on a machine configured that way OR that I won’t typo this one on a file that really matters.
I kinda use of `cat` as a `immutability-please` command: it _creates_ a barrier/seam between this file and whatever I’m doing with it, such that I’m _guaranteed_ nothing downstream will mutate it (as long as everything only works on stdio).
In a kinda similar way: by starting with `cat x |` (or even just `cat x`, frequently!), the rest of that compound command is _input source independent_ (and even _ignorant_!) — if I want to hook it up later to a `curl example.com/xyz.json`, I don’t need to split it into separate commands/introduce temp files/etc. This is especially because I find myself doing a _really_ quick scan of the entire file (“did it return the actual object this time or just `{}`?”), then piling on transforms from there.
Something about doing `cat x | head` and then kill-wording to fill in `next-thing` feels faster and more consistent than `head x` and then `next-thing x`, maybe because the filenames are often much longer than the commands? Maybe I need a “go to start of line, forward-kill one word” hotkey?
tl;dr: i’m playing psychological games with myself and TBH not sure I’m coming out ahead.
Wow, you described my workflow exactly. Using “
‘grep
This is why I walked back my claims a bit in my tldr — basically started trying stuff out and had a “whoa not used to the way that looks, but it does _work_…” moment.
I think I still have the “wrong” preference, but I appreciate learning some new stuff!
cat was just to demonstrate using it on streams. A more common use case would be to pipe some curl to it to see some API output, or grep some log file.
These lists always delight and frustrate me at the same time. Every piece of software seems to have a lot of those useful, little tricks, shortcuts and functionalities but discovering them is always so difficult.
Biggest problem is that you don’t even know they’re there. You don’t even know to look, what to look for. I seem to always stumble my way ass backwards into them, and then, be somewhat irritated that I didn’t know them before.
Hey that’s my old roommate!
Check out his vnlog project as well, it’s kinda feedgnuplot 2.0
How in the world did I go this long without using the “at” command? (slide 69)
Thanks for the deck, OP.
Wow. Well, I wish I would have known abut GNU parallel before I wrote that massive bash script a few years ago. Thanks for posting this.
Learning more unix tools tends to lead to that feeling a lot. The last few decades of devs building tools generally means that most obvious problems have a solution, but of course one usually only learns about that solution after hacking together their own version:) One tip: Look up moreutils; it’s a nice consolidation of things that make a lot of things easier. (One gotcha: moreutils ships its own different parallel tool.)
Its a very handy utility. Sometimes when I am lazy to write proper multiprocessing support in my codes, I get some work started anyway by (1) quickly providing a basic CLI version of my code (2) throwing in a bunch of code-invocation commands in a file and calling GNU parallel on it.
xargs can do most of the common tasks perl parallel does, see xars -P
Imho, the main advantage of Parallel is that it can execute the tasks on multiple hosts, straight from the man page:
> GNU parallel is a shell tool for executing jobs in parallel using one or more computers. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables.
It’s an absolute godsend for plowing through data-parallel computations.
In native Mac text editors, you can use e.g. SHIFT + ALT + LEFT to select the whole word on the left of the cursor. Is there a way to get a Mac terminal that uses similar shortcuts? For copy/paste also?
Related, but I found https://micro-editor.github.io/ which tries to use shortcuts you’d expect in native Mac apps.
There’s a couple options here: most of the Mac terminals allow you to specify custom key-combo -> Terminal input mapping that can be used to get this. Also, on most shells you can customize the interpretation of the key bindings to attach this key combo to a specific editing command. Which works best depends on the functionality and the key combo you’re trying to bind.
slides 14 and 15 – terminals have emacs shortcuts??!
EDIT: I am god now
You can configure readline (which is what bash uses) to use vi-bindings via $HOME/.inputrc then everything that uses readline (which is pretty much everything) will also get vi-bindings.
The main thing I miss from OS X is Droplr (pain free sharing of annotated screenshots) and Jumpcut (pastebuffer history tool). Can’t find anything on Linux for either of those that works nearly as well. I can still go to an old Mac for occasional needs like Final Cut Pro but those are tools I used several/dozen of times every day before.
I’m on PopOS.
I had not, just installed it though and it looks fantastic, thanks! The best I’d found previously was a no-annotation script that failed most of the time with a vague “something went wrong” error.
I hunted around for a while looking for something similar. I spent so much time in Slack working remote, that fast screenshots with arrows and notes are critical to communication.
Coming from a Mac environment, I can’t remember what the tool was that I used previously but I think it was purchased by Evernote. Flameshot has been excellent as a replacement.
You’re thinking of Skitch, I used that too before Droplr.
I have been using shutter and kazam for screen recording… but will check this out…
flameshot is excellent, but I think X11 only?
What about wayland?
Have not. Never heard of the Vala programming language before either. It does not seem to be a particularly popular application so definitely the kind where I would read the source first. Luckily it does seem pretty readable, so I will check it out, thanks!
Clipboard managers are probably the thing I miss the most on Wayland.
This is great! I might add a slide or two about Makefiles. Probably the topic is too big and out of scope, but inexperienced Unix users may not realize that they are useful for so much more than compiling multi-file C programs.
OP here — thanks for the comment. I actually had 2 slides dedicated to “program development tools” that included gcc tools and makefile. Had to remove them due to time constraints — had only 90 minutes to talk.
Please lets not recommend anyone new to Makefiles. It’s such a dated system that you need to use tabs for functionality.
Just one of million better alternatives like https://www.pyinvoke.org/
Makefiles need to be allowed to die so we can clear ourself of this absurd legacy cruft.
Make is for tracking dependencies between targets, and rebuilding stuff. Looks like pyinvoke does some glorified scripting, or something. I will never understand why so many people have no idea how to use Make, but are convinced that it is terrible, and then try to fix it by reimplementing some tiny subset of its functionality.
Make is great. Learn it an use it. Its limitations are well-known, but easy-enough to work around. Fixing them properly requires lots and lots of complexity, and you get bazel when you try. And it’s way overkill for all but the biggest projects, which is why Make has been so successful.
It’s was great but then we improved our work flow patterns. Make is really limited and ugly by today’s standards. I mean you have to use tabs for Pete’s sake and if you include a single space the program will break. It’s dated software that should be abandoned with either evolution or revolution.
Other useful tools include mosh or EternalTerminal to handle either low bandwidth or spotty network when you need to remote into a server, as well as VNC for when you need a GUI on the server but X-forwarding is too slow.
I expected something more of a traditional set of “productivity tools”.
Eg. a tool like “PDF Mod” is a gem I struggle to remember the name of when I set up a new computer — quick and easy way to assemble/extract PDF pages 🙂 It’s long unmaintained it seems, though.
For those expecting the same, this is mostly “GNU terminal tools” (+ python, tmux and a few other tidbits), so applies to pretty much any modern GNU system (including non-Linux systems, and I am pretty sure homebrew on Mac has all of these).
Thanks, these look great. Is there a recorded talk for these slides?
Unfortunately no. The conference recorded only regular track talks. This was on Tutorial/Training track.
Thanks for the comment. An example of #tag in action is shown in slide 39 (In the talk, I mentioned we will see an example later).
The only thing that I miss in linux is a pdf viewer that can show inline animations (like those produced by LaTeX’s “animate” package). I am still forced to run adobe’s 32 bit pdf viewer because it is the only viewer available in linux that supports this kind of pdf.
Wow, that seems like a rather questionable feature for a “portable document format”.
If you use pdf for making slides for a presentation, it is a legitimate use case to want some automatic animations or videos. It is actually possible, and adobe’s viewer certainly supports it, but no program on linux does.
You could argue that we shouln’t be using pdf for making slides in the first place, and maybe you are true. But if I want beautiful math typography (which is even more important than animations), all other options are sadly not up to par.
I understand; I used Beamer back in college. It just seems like that’s not really a good fit for the original purpose of PDF files. If they wanted to support presentations, maybe they should have created a new file type based on PDF but with extra capabilities.
I just tried it in Okular (KDE’s own) and it works. You have to let Okular “Show Forms” first, but it very conveniently shows the button at the top of the window.
Oh, man, you just saved my life! Now I can do everything I need with free software (notwithstanding that Okular’s interface is quite wonky, but whatever).
I never know about pdf inline animation. Do you have some sample of pdf files that contain inline animation? I would like to try them on various pdf viewer.
The documentation of the “animate” package contains an example on the first page:
Notice that, while this is just a silly example, animations are very useful in serious technical presentations. For example, in research about image or video processing, you often want to alternate automatically (without user interaction) between the “before” and “after” images, or along a few frames of a video sequence.
This is an awesome intro, thanks! Will definitely share.
I can’t recommend fish-shell highly enough for anyone spending significant time in the CLI. It can’t completely replace bash et al, but it’s not really meant to. Just install it and give it a try and you’ll get addicted pretty quickly.
I absolutely love this sort of thing. I am a sysadmin with something like 30 years experience. I am as passionate about the subject as anyone but I lack some presentation skills and possibly imagination. I’m also old enough to know what I lack!
Why not give open source stuff a go?
Some essential stuff lacking here (unless I missed it): pushd, popd, dirs, jobs, and fg.
I did have jobs, fg, nohup in an expanded version but had to trim to fit the slot. I considered pushd, popd but decided against it as it requires a bit of explaining — figured I won’t have much time.
I see. I tend to recommend zsh, anyway, which has navigational features that superannuates pushd, popd, and dirs.
Awesome! Thank you for sharing this. I’ve been trying to get better with my Linux CLI knowledge, and this will really help.
Is the presentation this was delivered to recorded? I’d watch whatever lecture this was from.
