I’m used to multiline comments being “/* */” (php, sql etc) or triple quotes (ala python) etc. I’ve been learning ruby lately and I’m just not really finding the “=begin…=end” syntax as “pretty” as some of the rest of Ruby has been.
This really isn’t a detriment, just found it odd that they handle them that way. I see most folks just use hash symbols with column editing features in text editors though.
Part reminder for myself to build some of this at home for playing around with, part for friends who are already doing this stuff. I’ll need to be somewhat generic here for obvious reasons.
I’ve been working with the development team to get all of the operations scripts (mostly bash but also some python, perl, ruby and then some PL/SQL as well) into a version control system. We decided to go with Git and because we’re already using Jira, Confluence, Crowd and a number of other Atlassian tools, we decided to go with Atlassian Stash.
Stash is kind of like Github or bitbucket for companies that don’t want to have their code hosted externally. It lets you create your own projects and permissions based at the project and repo level). With yesterday’s 2.4 release of stash it became a little easier because of the addition of repo level permissions (they were only at a project level from 2.3 and earlier).
The goal for the actual environment is to start with some basic configuration files, get them in a organized structure and then symlink to them. They’ll be owned by a new configuration user and then the only way to write changes to them will be to clone the git repository and go through the ‘pull request’ workflow. Then one of the senior members, such as myself, will approve that. Then we’ll use a tool called Jenkins (sometimes you’ll see it as its older version named hudson) to actually do the deployment. This gives us an audit trail and a way to control the execution. In time, all of our shell scripts will work in this manner.
The other project that I’m working on is log aggregation, but I’m still doing some testing/research there. The solution that I like the most so far is logstash along with some of the tools people use most often with it. In a nutshell it will aggregate all your logs (which helps with disk space on servers too) to a single server where they are consumed and then made available through an interface like kibana or greylog and you can do analytics on them. We’re hoping to do this with our application logs.
I really wanted to use/learn Nagios as part of this project but since I don’t have admin/root access it’s tough to push that project through another group as it’s just R&D right now.
A friend had asked me about an easier way to send all the output from a bash script to a log file. He said he was sick of doing something like:
logfile=/full/path/to/log echo "Some message" >> $logfile grep pattern file >> $logfile
Those are simple examples, but having to put the >> $logfile into each line could be a pain and then you might miss things on commands where you weren’t redirecting the output; this would be applicable for workload automation (cronjobs, autosys).
Bash has a built in named ‘exec’ that can lessen this load for you (among other things):
logfile=/full/path/to/log exec >$logfile 2>&1
To explain, the exec command is taking over the current shell running it and then sending stdout and stderr to this file. Now any command/function etc that is run within the script will send its output to that file.
Edit: I though I should mention that I use 2>&1 above because I end up getting stuck working on systems with older versions of bash. In modern cases this has been replaced with “&>”
Three years ago…
I should have gone all technical on this and just installed perl on my router that’s running linux, but I didn’t. I was trying to figure out something remotely going on with my connection so that my ISP would agree that there were some connection issues that they needed to address. Since my router is obviously directly connected to the modem, it was a good a candidate as any and I could remotely access it via SSH already.
I set a ping to run against google on it for 24 hours. Standard packet size and all that. I had all of this going out to a file (the router has some storage on it). I retrieved that file and wanted to parse it. I did some quick searches in the normal places (at that time) and didn’t find anything that really suited what I wanted so I did this in perl. I could have done it in bash. I’m not even sure why I used perl. I don’t use it often enough to be really proficient in it but I can read it and accomplish tasks in it when needed. Maybe I was feeling nostalgic. Comcast had told me for the purposes of support that for pings anything under 500ms (ugh) was acceptable to them and I had to prove it otherwise.
So you take a ping file (let it run however long you want. Pipe it through cut to get the relevent lines “cut –d= -f4 source_ping_file >> cut_ping.txt) and then run it against this.
Really, I’d have liked to have done it all in one perl script using some CPAN libs that did the pinging directly and all that, but I digress – it was a one-time-use thing and the goal was getting Comcast to help me out (which they did!).
If I were doing this for work…I’d have handled it differently for sure.
#!/usr/bin/env perl
#Script will take a ping file (ping someaddress.com >> pingfile.txt) from
#DD-WRT Routers and parse out the response times
use strict;
#Init Vars
my ($average, $total, $counter, $under50, $over500, $over600, $over700, $over800, $over900, $over1k, $over1100, $over1200) = 0;
#Logic
open pf, "cut_ping.txt";
foreach ( <pf> )
{
chomp($_);
#Get some idea of the range of where the issues are - could be cleaner if implementing the range module
if ($_ < 50 ){$under50++}
if ($_ > 500 ){$over500++}
if ($_ > 600){$over600++}
if ($_ > 700){$over700++}
if ($_ > 800){$over800++}
if ($_ > 900){$over900++}
if ($_ > 1000){$over1k++}
if ($_ > 1100){$over1100++}
if ($_ > 1200){$over1200++}
$counter++;
$total = $total + $_;
}
$average = $total / $counter;
#Script Output
print "Total of " . $counter . " lines is " . $total . "\n";
print "Average Ping time is " . $average . "\n\n";
print "The ping time was under 50 " . $under50 . " times\n";
# test over 500 but lower than 600
my $b500_600 = $over500 - $over600;
print "The ping time was between 500ms and 600ms " . $b500_600 . " times\n";
my $b600_700 = $over600 - $over700;
print "The ping time was between 600ms and 700ms " . $b600_700 . " times\n";
my $b700_800 = $over700 - $over800;
print "The ping time was between 700ms and 800ms " . $b700_800 . " times\n";
my $b800_900 = $over800 - $over900;
print "The ping time was between 800ms and 900ms " . $over800 . " times\n";
my $b900_1k = $over900 - $over1k;
print "The ping time was between 900ms and 1000ms " . $over900 . " times\n";
my $b1k_1100 = $over1k - $over1100;
print "The ping time was between 1000ms and 1100ms " . $b1k_1100 . " times\n";
my $b1100_1200 = $over1100 - $over1200;
print "The ping time was between 1100ms and 1200ms " . $b1100_1200 . " times\n";
print "The ping time was over 1200ms " . $over1200 . " times\n";
For work right now we do most of our stuff over SSH. We’re issued laptops at work that have windows on them (no say in the matter) so generally we’ll run VMs if we need to be as linuxy as possible. Since most of it is command line work over SSH I will usually just use Putty and connect there and do my thing.
So Putty itself is awesome and there are lots of projects out there that extend the functionality even more, like Kitty. So how do you get the most out of putty? Some of the downsides (just my opinion!) are how it organizes the information for your connections and then you have to deal with a separate window for each connection (not exactly ideal).
A few years back I had started to use Mremote (before the NG project) and found that it gave me what I was looking for:
TL:DR; If your 13.04 install looks like the wireless is connected, you can ping after connecting but cant after a little bit of time and you have a broadcom wireless card that identifies as pci.id 14e4:4311 (lspci -nn) – keep reading.
I installed Xubuntu 13.04 this morning on my laptop (Dell Inspiron 1564, Core i3, 4gb Ram etc etc). I noticed during the installation that it was hung up trying to download fresh packages so I just skipped that part and let the installer do its thing and figured that I would just update later.
It wasn’t necessarily a surprise to me that the wireless might not work right away as I’ve had to do similar adjustments in other OS’ like Fedora or Mint.
Well…this one was a little different than just plain old “not working.” Initially it would connect, you can ping, visit sites…and then about 5 minutes later you can’t really do anything network-wise. You can ping localhost, but not even the router/ap it says you’re still connected to.
“ip addr” show you have a valid IP and netmask and all that.
So first I verify the wirelss card
josh@yardlobster$ lspci -nn | grep 0280 05:00.0 Network controller [0280]: Broadcom Corporation BCM4311 802.11b/g WLAN [14e4:4311] (rev 01)
And I figure I just need to install the right driver like I’ve had to before, and it’s usually in the repos anyways. I plug in a cable and then try to install “bcmwl-kernel-source” like I normally do to fix this. But Ubuntu’s all like “Hey Bro, I’ve already got that, and it’s the latest version.” So I figure I’ll remove –purge and reinstall, which completes but doesn’t solve the issue after a reboot.
I start googling at that point and start seeing some similar issues posted in ubuntu 13.x beta support threads etc. One of them suggests to install “linux-firmware-nonfree.” I figure I want to understand what all is included…and after an hour of looking through gentoo, arch, debian, mint and ubuntu posts and threads all I can see is people recommending to install it (mostly for TV tuner cards) but no info on who maintains it or the depth of what’s in it. I’m going to continue to track that down, but it does solve the problem.
josh@yardlobster$ sudo apt-get remove --purge bcmwl-kernel-source ... josh@yardlobster$ sudo apt-get install linux-firmware-nonfree ...
Ahh, good ol dd…I think I’ve heard someone effectually refer to it as ‘disk destroyer’ at some point in my career. At any rate, I already had an ISO I wanted to put onto a USB drive as bootable so I could install an OS from it.
dd if=/full/path/to/file.iso of=/dev/sdb
Some caveats:
You can also just grab Unetbootin from sourceforge (http://unetbootin.sourceforge.net/) and use it to play with either ISOs you already have, or just have it figure out where to get the image you want (it has some popular distro sources baked in already).
This is the easier to understand solution of the two but not always an option, especially if you’re working from a headless system.
I’ve been evolving my home workstation for a while. Here’s the current incarnation:
Hardware:
On the wireframe rack to the left is my Linux Laptop (Core i3, 4gb Ram, 250gb HD) and a couple of netbooks I use as dummy clients for testing.
To the right is the linux server that my family uses for backup/fileserving (Samba), Media Server (PS3MediaServer), and as the simple download provider for epub books (apache), and then some web based control apps to make things simpler for the family (restart certain services like the media server).
I have an environment on a Redhat machine that I have to share with about 7-8 other people. We SSH to the machine and then su over to a specific user in order to perform work as that user (permissions are different etc). It’s not exactly ideal, but it works.
One of the problems is that the environment actually resides on a NAS that’s mounted on various machines that people could potentially be signing in from. If you’ve customized or looked into the inner workings of bash history before (if not – here is an excellent primer: http://www.symkat.com/understanding-bash-history ) you know that the history writes on session close, and it’s going to overwrite anything saved to it in the interim by any other session.
With 8 people, bash history basically becomes worthless at that point. Understanding that each user was signing in as themself and then su’ing over to the shared user, I knew we could make this
better.
What you need to take into consideration as well is if you have any non-interactive connections that occur (other machines SSH’ing in via automation). We definitely do, via autosys so the approach I took was specific to our environment.
It was really as simple as this:
export HISTFILE="/home/svbatch/.bash_history-$(who am i | cut -d ' ' -f1)" shopt -s histappend #never clobber history, always append export PROMPT_COMMAND='history -w' #executes this command immediately after any given command, appending to the history instantly for tailing across systems that share a NAS export HISTTIMEFORMAT="%Y%m%d-%T "
The real trick is setting the histfile to a unique filename upon login. “who am i | cut -d ‘ ‘ -f1″ accomplishes this, however simply changing this to ‘$(logname)’ would have done the trick too (which is actually what I’ve changed mine to on the server in question). The other options set it up so that after every command is run, history appends rather than clobbers over.
Some other tricks you can do with logname in a shared environment are loading custom shell functions/alias files
#attempt to load custom functions if they exist
userfunctions=/home/shareduser/.functions/$(logname)_functions.sh
if [ -f "$userfunctions" ]; then
source $userfunctions && echo "[ OK ] Sourced $(logname)'s functions file";
fi
Built this after my wife got me a new electric guitar for my birthday (I hadn’t played electric in maybe 6 years at that point)…and then I got a 12 string with some more of my birthday money.
I used rubber coated sledgehammer hooks from the local ACE hardware store. I used four 100lb wall anchors and then used a countersink bit to make sure the screws were cleanly countersunk.
