virtual insanity

 

Given my recent inactivity here and on Twitter, I feel the need to post some updates.  So, where have I been and what have been doing for the past few weeks?  I’ll start with the most recent drama, which should give you a good laugh.

Fractured Sternum

A few months ago, my wife and I decided to buy our four year old son an outdoor playset.  Our local Costco had the “Rainbow All-American Double Decker Playset” (pictured left) on sale, so we decided to buy it.  That was back in March.  Nearing the end of June, do you know where the playset is?  Still boxed up in our garage.  Some Dad I am. 

Anyway, on Monday I decided I was sick and tired of all the clutter in our garage.  Plus I was determined to get that playset built before the end of June.  So I decided to rearrange the garage and get the playset boxes ready to be moved to the back yard.

Before I continue, let’s take a quick look at the weight and dimension of these boxes …

Shipping Box Dimensions:

• Box-1: 14 1/2” L x 11 1/4” W x 9” H: Approximately 40-lbs

• Box-2: 22 1/2” L x 11 1/4” W x 9” H: Approximately 40-lbs

• Box-3: 106” L x 24” W x 7” H: Approximately 210-lbs

• Box-4: 106” L x 24” W x 7” H: Approximately 240-lbs

• Box-5: 106” L x 24” W x 7” H: Approximately 195-lbs

• Box-6: 106” L x 24” W x 7” H: Approximately 200-lbs

• Slide: 115 1/2” L x 24 3/4” W x 16 3/4” H: Approximately 40-lbs

 

Well, as I was trying to be the big, bad, super dad and move Box #4 on my own … I had a little accident.  That’s right, I have a fractured sternum because a playset box fell on my chest!  How embarrassing.  My friends affectionately now call me “crash” and my wife will no longer allow me to go into the garage without first showing her my helmet is securely fastened.

The good doctor from the ER gave me some Vicodin for the pain, which has been very helpful.  But the side effect is that Vicodin makes me loopy, making it difficult to write.

VCDX

If you’re still reading then you’re probably questioning my level of intelligence (and I wouldn’t entirely blame you   So I figured I would try to redeem myself with an update on my VCDX progress.  Even though I have not yet posted all of my VCDX study notes, I actually took the VCDX admin exam a few weeks ago.  And I just found out last week that I passed!  Woooohoooo!  Now I’ve got to start preparing for the next test, the VCDX Design Exam.

 

VMworld2009

Looks like I’ll be speaking at VMworld2009.  So if you’re planning to attend this year’s event, be sure to say “hello.”  Just look for my shaved noggin’ wondering the halls (or the guy with the shiny helmet, hehehe).  Even better, as I’ll be the speaker of session DV3567, you can certainly find me at the following breakout session …

Session ID:
DV3567

Title:
Don’t throw that PC away! How to convert old PCs to Thin Clients using a thin Linux OS and VMware View Open Client.

Abstract:
More and more, companies are looking for additional ways to cut costs though virtualization. And it isn’t long before IT teams start exploring the possibility of a Virtual Desktop Infrastructure. But with desktops out numbering servers by a factor of 10:1 (or more), converting users to a virtual desktop can be technically challenging and a significant upfront expense. A potential solution to this problem is to convert existing PCs into Thin Clients, extending the life of the hardware and easing the transition into a VDI. This session will show IT professionals various ways to convert older PCs into Thin Clients, capable of connecting to a VMware VM hosted on ESX via the VMware View Manager.

RoR (Ruby on Rails) and other next generation frameworks

I like to think of myself as an amateur developer (though, even amateur developers might have a thing or two to say about that!!   I began programming in Perl about 10 years ago and since then I’ve dabbled in a number of different languages, like C++, Java and Ruby.  

About two years ago I was introduced to Ruby on Rails and since then, most of my development work has been with RoR.  Thus far, however, I haven’t posted anything on this blog about RoR.  Why?  Two reasons.  The apps I’ve written to date have absolutely nothing to do with VMware.  And second, like I said, I’m an amateur.  Anyone looking for RoR help and advice can probably find better info on actual RoR blogs.

But I’ve decided that this is about to change.  Most recently I’ve been working on a little RoR front end that will “drive” vSphere via SOAP.  So I certainly find that work relevant here.  Plus, if you think about it, Rails provides a level of abstraction and therefore, by definition, can be called a type of virtualization. 

So if you’re an RoR developer (or any other kind of next generation framework, for that matter), please let me know.  I’m interested in reading your blog, checking out your applications, sharing code, chatting about issues / concerns / challenges, etc.  Just post a comment or email me at asweemer [at] gmail [dot] com.

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

Ruby on Rails, VCDX, VMware, VMworld 2009

Just noticed that ThinApp 4.0.3 was released tonight and available for download.

Release notes: http://vmware.com/support/thinapp4/doc/releasenotes_thinapp403.html

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

Miscellaneous

OK, we are *almost* done getting our network set up properly for VDI, but we’ve got a few more things to do.  Specifically, we need to address:
1. Handling our external, Internet facing, dynamic IP address.
2. DHCP and DNS
3. External VPN access

Dynamic External IP

Most ISP’s (not all) provide a single, dynamic IP address for consumer grade service (i.e. home use).  But when we’re trying to connect to our virtual desktops from somewhere out on the public Internet, how do we know which IP address to connect to?

Generally speaking, connecting to an IP address is bad practice because it’s inflexible.  Instead, we should connect to a Fully Qualified Domain Name (FQDN).  OK fine, so we’ll set up a DNS entry and use a FQDN to connect to our desktops.  But what happens when our dynamic IP address changes and the DNS entry is still mapped to the old IP address?

What we need is an external Dynamic DNS (aka DDNS) service which will allow us to programmatically update our IP address whenever it changes.  There are a number of both free and paid-for DNS providers out there that can deliver DDNS services.  Personally, I use EditDNS (www.editdns.net).  They have a ton of functionality and they’ve been rock solid for the past few years I’ve been using their services, so I’m quite happy with them.

Now, many home use routers these days have the capability to update a DDNS provider.  But in my experience, the functionality is somewhat limited.  What if, for example, I want aaron.sweemer.com and desktop.sweemer.com to be dynamic entries and www.sweemer.com to be a static entry, pointing to my blog server hosted somewhere else?  In reality, I’ve got about 20 FQDN’s that I need to be dynamically updated and about 100 that I want static.  So instead, I created a script that will:

1. Query my external IP address (check this out, a free tool from Whatismyip.com)
2. Compare the result of the query with the IP obtained from the previous query
3. If the IP is the same, or contains something other than an IP (e.g. HTTP error), the scirpt exits.
4. If the IP is different, the script updates my DDNS entries via the EditDNS API, then updates a log file documenting the change, and finally adds the new IP to the last line of a file called previous_ips.

If you’d like to use the script I wrote, you’ll first need to do the following:

1. If don’t already have one, set up an account with EditDNS and make sure you have properly configured the domain name(s) you own.
2. Verify your linux distro has lynx (a command line, text only, www client)
3. Verify your linux distro has curl (a tool to transfer data using HTTP)
4. Create a directory (anywhere you have rwx access is fine) for the script and its files to live
5. In this directory, create a text file called editdns.sh.  Paste the content (below) into it.
6. Replace XXXXXXXX with your EditDNS password.
7. Make editdns.sh executable (chmod +x /path/to/editdns/editdns.sh)
8. Create another text file called records and, one per line, enter the FQDN’s of the DDNS entries you wanted updated (e.g. mydesktop.mydomain.com)
9. Add the editdns.sh script to your crontab to run at regular intervals (e.g. mine runs every five minutes and the entry in cron looks like this:    */5 * * * * cd /usr/local/editdns; ./editdns.sh)
   And here’s a copy of the actual script …

#!/bin/bash

EDITDNSPASS=”XXXXXXXX”
LYNX=`which lynx`
TIME=`date`
CIP=`curl -s http://www.whatismyip.com/automation/n09230945.asp | awk –re-interval ‘$1 ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/ {print}’`
PIP=`tail -1 ./previous_ips`

if [ "$CIP" != "$PIP" && –n "$CIP" ]; then
cat ./records | while read FQDN; do
$LYNX -source “http://DynDNS.EditDNS.net/api/dynLinux.php?p=$EDITDNSPASS&r=$FQDN”
done
echo “IP Change!  New IP is $CIP.  Editdns.net was updated at $TIME.” >> ./editdns.log
echo $CIP >> ./previous_ips
else
exit 0
fi

If you have any issues or questions, feel free to email me.  Also, keep in mind, this a quick and dirty script that accomplishes what I want it to accomplish.  Feel free to make it more robust (e.g. error handling or better logging) to suit your needs.

DNS and DHCP

It’s quite possible you’ll want to skip this section and opt for setting up DHCP and DNS via Microsoft’s built in DHCP and DNS services that come out of the box with their server products.  To properly set up VMware View, we’ll need to set up Active Directory anyway, and quite frankly, it’s far easier to set up a Microsoft server with DHCP and DNS than it is to set up a Linux server.  So feel free to skip this section and leverage Microsoft for these services.  If, however, you’re a gluten for punishment then by all means, read on.

Let’s first start with DNS.  Here too we need Dynamic DNS because as we’re handing out IP addresses via DHCP, we want our DNS server to properly reflect current information as IP addresses change.  So, if you don’t already have bind9 (the DNS server), go ahead and install it (sudo apt-get install bind9 should work on Ubuntu / Debian distros).

The default configuration for bind9 is to act as a caching server, so the first thing we need to do is configure our DNS to forward all unknown DNS requests to another DNS server.  These should be provided to you from your ISP.  Edit the forwarders {} section of your named.conf.options file (usually located in /etc/bind/) to look like this …

asweemer@cincylab-rtr1:/etc/bind$ more named.conf.options
options {
directory “/var/cache/bind”;

forwarders {
1.2.3.4;
5.6.7.8;
};

auth-nxdomain no;    # conform to RFC1035
listen-on-v6 { any; };
};

Obviously, you’ll need to change 1.2.3.4 and 5.6.7.8 to the IP addresses given to you by your ISP.

Next, we need to modify our master named.conf to allow dynamic updates to DNS.  Add the following entry to the bottom of your named.conf file.

controls {
inet 127.0.0.1 allow {127.0.0.1; 192.168.9.25; 10.10.7.1; 10.10.7.2; } keys {”rndc-key”;};
};

This tells the DNS server to allow updates from the IP address located between the {}.  Notice the first three IP addresses are local IP addresses.  The fourth IP address is a slave DNS server, which I have yet to set up.  The rndc-key is the default key generated during installation of bind9 and it’s used to authorize the updating of DNS records.  If you’re using Ubuntu, then you’ll likely find the key in the file /etc/bind/rndc.key …

asweemer@cincylab-rtr1:/etc/bind$ sudo cat rndc.key
key “rndc-key” {
algorithm hmac-md5;
secret “QZ5jOmcr/OW3nzksR5q0Hw==”;
};
asweemer@cincylab-rtr1:/etc/bind$

Note the file is a text file named rndc.key, and the actual key is called rndc-key located within the text file.

OK, next we need to define our zones in the named.conf.local file.  For each domain you’re using (probably just one), you’ll need two entries:  one for the domain and one for the reverse lookup of the domain.  I have two domains I’ll be updating, so my named.conf.local file looks like this …

asweemer@cincylab-rtr1:/etc/bind$ cat named.conf.local
//
// Do any local configuration here
//

include “/etc/bind/rndc.key”;

zone “mydomain.com” {
type master;
file “/etc/bind/zones/mydomain.com.db”;
allow-update { key “rndc-key”; };
allow-transfer {10.10.7/24; };
};

zone “7.10.10.in-addr.arpa” {
type master;
file “/etc/bind/zones/rev.7.10.10.in-addr.arpa”;
allow-update { key “rndc-key”; };
allow-transfer {10.10.7/24; };
};

zone “dmz.mydomain.com” {
type master;
file “/etc/bind/zones/dmz.mydomain.com.db”;
allow-update { key “rndc-key”; };
allow-transfer {192.168.9/24; };
};

zone “9.168.192.in-addr.arpa” {
type master;
file “/etc/bind/zones/rev.9.168.192.in-addr.arpa”;
allow-update { key “rndc-key”; };
allow-transfer {192.168.9/24; };
};

A couple points to note here:

• I created a subdirectory called “zones” under /etc/bind/ where I put all my zone files.  This isn’t the default location, and in addition, this isn’t necessary as the zone files can be located anywhere you’d like.  But be aware the configuration file above reflects the location of my files.
• Notice the include “/etc/bind/rndc.key” on the first line and the all-update directive within each zone definition?  This should be self explanatory at this point.
• The allow-transfer directive within each zone definition explicitly limits zone transfers (copy) to the IP(s) defined.  This is an important security feature since, by default, DNS allows transfers to anyone, and the info contained within a DNS zone file can really give hackers visibility into your network.

Now we need to create the zone files we just defined above, which will contain our actual DNS records.  Here is the zone file for our dmz.mydomain.com …

asweemer@cincylab-rtr1:/etc/bind/zones$ cat dmz.mydomain.com.db
$TTL 3600       ; 1 hour
dmz.mydomain.com             IN SOA  master.dmz.mydomain.com. root.master.dmz.mydomain.com. (
2009060514 ; serial
86400      ; refresh (1 day)
86400      ; retry (1 day)
2419200    ; expire (4 weeks)
3600       ; minimum (1 hour)
)
NS      master.dmz.mydomain.com.
NS      slave.dmz.mydomain.com.
A       192.168.9.25
MX      10 mail.dmz.mydomain.com.
MX      20 mail-spool.dmz.mydomain.com.
computer-1              A       192.168.9.247
TXT     “317bf41a2c5b70fd9ca4e283d364dcddd5″
computer-2              A       192.168.9.250
TXT     “00cf6242f693ebbf1d545159548e44ab81″
computer-3              A       192.168.9.243
TXT     “31a0cb7e096a96c63dc998d2db3be6e450″
mail                    A       192.168.9.25
mail-spool              A       192.168.9.26
master                  A       192.168.9.25
www                     CNAME   master

A couple important things to point out here:

• The entries for computer-1, 2 and 3 are dynamic entries there were generated by the DHCP server.  The TXT record that follows these entries is a unique identifier which is also generated by the DHCP server and is used to ensure it won’t overwrite existing DNS records that were generated by another process/server.
• You’ll obviously need to change the domain names and IP addresses to match your environment.
• If you haven’t worked with bind9 before, this file probably looks pretty cryptic to you.  If so, I would recommend taking a look at http://www.zytrax.com/books/dns/ch8/soa.html, which gives a pretty good overview of the SOA (defined in the first part of the file).  The balance of the file (i.e. the record definitions) is pretty straight forward.

The reverse zone should look like this …

asweemer@cincylab-rtr1:/etc/bind/zones$ more rev.9.168.192.in-addr.arpa
$ORIGIN 9.168.192.IN-ADDR.ARPA.
$TTL    1h;
IN      SOA     master.dmz.mydomain.com. root.master.dmz.mydomain.com. (
2009060501      ;
1d              ;
1d              ;
4w              ;
1h              ;
)
IN      NS      master.dmz.mydomain.com.
IN      NS      slave.dmz.mydomain.com.
25      IN      PTR     master.dmz.mydomain.com.
26      IN      PTR     slave.dmz.mydomain.com.

I mixed it up just a bit in this file to point out a few different ways to configure a zone file.  In this file, notice the following differences:

• The $ORIGIN directive sets the domain name to be appended to any unqualified records.  If the $ORIGIN directive doesn’t exist (as it doesn’t in the first config file), then it is implicitly defined by the zone name.
• The time variables can be defined with d (day), w (week), h (hour), etc.

That’s about it for DNS.  Once you’ve got your bind9 server configured, restart your bind9 server (sudo /etc/init.d/bind9 restart).  And of course, be sure to test your configurations by using the standard DNS tools (e.g. dig, nslookup).  If you get errors, pay careful attention to your local syslog file (probably located at /var/log/syslog) as that’s where DNS and DHCP errors typically write their error messages.

OK, next up is configuring our DHCP server.  And once again, this post is starting to get way to long, so it looks like I’ll need a fourth and (hopefully) final post to this section.

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

Dynamic DNS, Linux, VDI, VMware, VMware View, e.t.d.f series

With the release of View 3.1 we received some more flexibility with presenting/hiding Human Interface Devices  (think foot pedals for a transcriptionist, some type of bardcode scanners, etc).

HID devices are filtered out by default as it would be a bad thing if your local mouse was redirected for example.  So to enable a specific device to be passed through we need to do a few things:

1:  First we need to determine the VID/PID of the HID device.  There are two ways to determine this:

a:  Debug Logs:  Go to C:\Documents and Settings\All Users\Application Data\Vmware\VDM\logs.  Search through the log for “Devices”  This will contain the information of all the devices available before filtering takes place

b:  Windows Device Manager:  Open Windows Device Manager and find the HID device you are interested in.  If you right-click–>Properties on the object and go to the Details tab you will see a drop down.  Choose Device Instance ID from the drop-down.  The VID/PID value will be displayed.  Usually this looks something like USB\VID_xxxx&PID_xxxx\…

2:  Now that we know the VID/PID we can go to the client and create the appropriate registry keys to tell the View Client to pass that particular HID device through:

a:  Go to HKLM\Software\VMware, Inc.\VMware VDM\USB\

b:  Create a new Multi-String Value named AllowHardwareIDs

c:  Set the value data to the VID_xxxx&PID_xxxx you documented earlier

d:  Restart the client and things should work upon the next connection

Special Thanks to Pete Barber for this info!

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

Miscellaneous

I got up early to get some work done before driving down to KY to work with a customer on their VDI pilot.  As I was preparing for the meeting, I thought to myself, “wow with my studies for the VCDX admin exam, and the recently launch of vSphere, I haven’t done a whole lot of VDI recently.”  And then BAM!  It hit me like a ton of bricks.  I haven’t completed the E.T.D.F series I started almost 6 months ago!  If this blog has done anything for me, it has made me painfully aware of my numerous character flaws. <sniffle><tear><sniffle>

Anyway, not that anyone is following along anymore, and purely in the interest of self improvement, I’m determined to finish what I’ve started (both for this series and my VCDX study notes series).  So without further delay, here is the second part of the networking section (here is part one).  And for your convenience, here again is the Visio diagram of my lab.

Router / Firewall (cincylab-rtr1)

In my environment, the vast majority of all relevant network configurations are on cincylab-rtr1, which is really an old Gateway PC that I had lying around with a single 2.2GHz processor and 1Gig of RAM and a single 100Mbps NIC.  I installed Ubuntu server 8.04.1 (kernel 2.6.24-19-server) on it and made it the gateway between my lab and the DMZ (aka, my home network).

The first thing I needed to do was get the basic networking on the server set up.  I have three networks in my house …

1. An external DMZ, VLAN 192 (aka, my home network)
2. An internal “production” network, VLAN 10.  I put the word production in bunny ears because nothing is *really* production … it’s all just a lab.  But I try to protect this network a little more than the next.
3. An internal “lab” network.  This is where I can really have fun!

Configuring the Interface(s)
The server only has one NIC and I was too lazy (and cheap) to go buy a new one.  But it’s a 1GigE card, which is plenty for my environment.  And it’s a snap to configure …

root@cincylab-rtr1:/etc/network# more interfaces

auto lo
iface lo inet loopback

auto vlan10
auto vlan192
auto vlan10:1

iface vlan192 inet static
address 192.168.9.25
netmask 255.255.255.0
gateway 192.168.9.1
mtu 1500
vlan_raw_device eth1

iface vlan10 inet static
address 10.10.7.1
netmask 255.255.255.0
broadcast 10.10.7.255
network 10.10.7.0
mtu 1500
vlan_raw_device eth1

iface vlan10:1 inet static
address 10.0.1.1
netmask 255.255.255.0
broadcast 10.0.1.255
network 10.0.1.0
mtu 1500
vlan_raw_device eth1
root@cincylab-rtr1:/etc/network#

Turn on Routing

Now that the interfaces are configured, we need to turn on routing.  In Linux, this can be accomplished a couple different ways.  The easies, IMHO, is to simply edit the /etc/sysctl.conf file and set net.ipv4.ip_forward=1.  You could also add echo 1 > /proc/sys/net/ipv4/ip_forward to your /etc/rc.local file.  Either way should turn on IPv4 routing on your server.

Configure NAT and PAT (Port Address Translation)

Once routing is turned on, we need to set up Network Address translation and Port Address Translation.  This needs to be done for two reasons.

1. My lab networks need outside access to the Internet and they have private IP addresses.
2. My server has a single IP address in the DMZ, which needs to serve as the gateway IP for multiple internal IP’s and TCP/UDP ports.  As an example, I want all traffic arriving on 192.168.9.25, TCP port 8080 to be forwarded to the internal IP 10.10.8.51 port 80.  And more specifically, here’s what I want available to the outside world …
   • 192.168.9.25:8080 –> 10.10.7.51:80
   • 192.168.9.25:8181 –> 10.10.7.51:443
   • 192.168.9.25:8282 –> 10.10.7.50:80
   • 192.168.9.25:8383 –> 10.10.7.50:443

OK, so how do we do this?   We need configure iptables.  An iptables tutorial is out of scope for this post, but if you’d like to learn more about Linux IP tables, I personally like this one:  http://www.yolinux.com/TUTORIALS/LinuxTutorialIptablesNetworkGateway.html.

To set up iptables, I’ve created a file called fw_rules in /usr/local/bin and made it executable (chmod +x /usr/local/bin/fw_rules).  Here is what the file looks like.

root@cincylab-rtr1:/usr/local/bin# cat fw_rules
#!/bin/bash
iptables -t nat -F
iptables -t filter -F

iptables -t nat -A PREROUTING -p tcp -i vlan192 -d 192.168.9.25 –dport 8080 -j DNAT –to 10.10.7.51:80
iptables -t nat -A PREROUTING -p tcp -i vlan192 -d 192.168.9.25 –dport 8181 -j DNAT –to 10.10.7.51:443
iptables -t nat -A PREROUTING -p tcp -i vlan192 -d 192.168.9.25 –dport 8282 -j DNAT –to 10.10.7.50:80
iptables -t nat -A PREROUTING -p tcp -i vlan192 -d 192.168.9.25 –dport 8383 -j DNAT –to 10.10.7.50:443

iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT

root@cincylab-rtr1:/usr/local/bin#

To make these changes persistent across reboots, you’ll need to add /usr/local/bin/fw_rules to your /etc/rc.local file.

Now, for all you linux experts out there looking at this file, you’re probably saying “uh, that’s a pretty insecure firewall you got there!.”  And you’d be right   Remember, this is merely an internal firewall/router which is protected by a much more secure, Internet facing, Cisco ASA (thanks again to the local Cisco team!!).  It’s also this Cisco that forwards outside connections on specific ports (not 8080, 8181, and 8282, for additional security) to IP 192.168.9.25.  And because of this, my goal for this server isn’t to protect, but to separate my lab networks from my home network, and proxy the connections between them.

What’s Next?
We’ve configured our networks, turned on routing and configured NAT / PAT on our server.  What next?  Three things:

1. Because my external IP is dynamic, we need to set up a script that will periodically check to see if our external IP has changed and, if so, update our dynamic DNS service.
2. Configure DHCP and DNS.
3. Set up external VPN access.

Step three is actually optional because when we’re done, we’ll be able to tunnel via SSL to our desktop.  And from our desktop, we’ll have full access to the local LAN.  But sometimes full remote access via VPN is nice without being forced to first “hop” to another desktop.  So, I’ll include my VPN configuration as well.

But for now, it looks like I’m going to need have a part three of this “setting up the network and dedicated remote access” section, because I need to get on the road down to KY.  But if you’re interested, look for part three later today.  I’m almost done with it and will try to finish it during my lunch break.

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

Linux, Ubuntu Linux, VDI, VMware, VMware View, e.t.d.f series

I haven’t had much exposure to KVM yet, so over the weekend I decided to check it out on my laptop.  After playing around with it a bit, I needed to power on an instance of VMware Workstation, and the following error popped up on my screen …

The virtualization capability of your processor is already in use.  Disable any other running hypervisors before running VMware Workstation.

Well that makes sense.  So I uninstalled KVM and the issue was resolved … or so I thought.  This morning as I powered on another instance of VMware Workstation, I got the same error again.  Hmmmm.  That was a bit more confusing because, to my knowledge, KVM was completely removed from my system.  Again, so I thought.  But a quick look at the currently loaded kernel modules revealed both the kvm_intel and the kvm modules.

As it turns out, when you remove KVM via apt-get (meaning, this *could* be a debian / ubuntu issue, not sure if other package managers do the same thing), it doesn’t actually completely remove itself.  The kvm and kvm_intel modules not only remain, but they continue to get loaded upon startup.  When I removed the modules, my VMware Workstation powered on without issue.

So then, that voice inside of my head — the one I should NEVER listen to — said “I wonder what happens when you load the KVM modules after you’ve powered on your VM?”  I *knew* it could only lead to bad things, but I couldn’t help myself.  Guess what?  Not only did VMware Workstation completely freeze, but now I can’t power on my VM, no matter what I try.   Grrrr.  I swear, someday I’ll be a news headline that reads … An eyewitness confirms his last words were, “I wonder what this button does?”

Anyway, if you get this error, simply check for the kvm modules (lsmod | grep kvm should do the trick).  Simply removing the modules will fix the issue.

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

KVM, Linux, Ubuntu Linux, VMware, VMware Workstation

As you probably have seen, View 3.1 GA’d yesterday.  One of the improvements listed in the release notes was:

• USB Improvements – View 3.1 offers more reliable and broader device support with reduced bandwidth consumption. A separate TCP/IP stream is used.

From what I understand in talking to some people is that a lot of time was spent on the USB redirection stack to further optimize and tune it.

ALSO, USB redirection traffic is now split out onto it’s own traffic stream.  USB redirection traffic will now communicate from the client to the host vm on TCP port 32111.  I imagine this opens up a few new opportunities to do some USB specific traffic prioritization/trottling.  Very interesting!   In previous versions, the USB traffic was inside of the RDP stream (virtual channel).  This prevented us from ever REALLY seeing the USB specific traffic or having any control over it.  Simply put, now we do.  Gotta love progress!

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

Miscellaneous

Frequently customers have specific NICs (like onboard NICs) that they’d like assigned to the COS, leaving the other NICs for VM traffic.  This is difficult, however, when using our automated kickstart deployment scripts as there is no way to explicitly define the vmnic assigned to the COS.  And to make matters worse, the VMkernel is not yet available to us during the %post section of the kickstart script, which makes COS networking configuration difficult! Recently I had a customer who was getting frustrated because …

1. They would “rack and stack” a physical server and wire up their NICs accordingly (i.e. onboard NICs on the management VLAN, remaining NICs on production VLANs)
2. PXE boot the server
3. When kickstart completed, they’d lose connection to the COS.

This happens because during installation, ESX just assigns vmnic0 to the lowest PCI number, and then assigns vmnic0 to the COS. And this is often not the NIC the admin wants used for their COS. Of course, they could go back after the fact and reconfigure the COS networking, but this kind of defeats the purpose of a completely hands-free, automated deployment.

Here is one possible solution to the problem.  Below is a script I wrote to append to the %post section of a kickstart file.  Obviously, you’ll need to make modifications for your environment.

## This script should be appended to the %post section of an ESX kickstart file. ## For more info on kickstart and scripted ESX installations, see Appendix B of ## http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_installation_guide.pdf ## ## Essentially, this is a “script that creates a script.” Because the VMkernel is ## not yet available to us during the %post section of the scripted install, we use ## %post to generate a script called /tmp/post_esx_install.sh that will launch via ## rc.local upon first boot (and only first boot). ## ## The post_esx_install.sh will first make a backup copy of esx.conf and then ## reconfigure the COS networking.  Please see the in-line comments below for ## tweaking post_esx_install.sh for your environment. ## ## If you have any questions, please email aaron [at] sweemer [dot] com. %post cat > /tmp/esx_post_install.sh << EOF #!/bin/bash cp /etc/vmware/esx.conf /etc/vmware/esx.conf.backup /usr/sbin/esxcfg-vswitch -U vmnic0 vSwitch0 /usr/sbin/esxcfg-vswif -d vswif0 ## If your kickstart file has vmportgroup=1, you *might* want to uncomment the ## next line ## /usr/sbin/esxcfg-vswitch -D “VM Network” /usr/sbin/esxcfg-vswitch -A “VMkernel” vSwitch0 ## You’ll need to find which physical NICs you want assigned to your COS.  From ## the command line of an already installed ESX server, execute ## “/usr/sbin/esxcfg-nics -l” as root and look for something unique about the ## NICs.  For example, this could be the word “Broadcom” or it could be the ## actual PCI number.  In the next line, replace “search term” with this ## text. /usr/sbin/esxcfg-nics -l | awk ‘\$0 ~ /search term/ {print \$1}’ | xargs –n 1 /usr/sbin/esxcfg-vswitch vSwitch0 –L ## Note: if you want to test the line above from the command-line, you’ll need ## to remove the leading “\” in front of $0 and $1. The \’s need to be here so ## the esx_post_install.sh script gets properly written by kickstart. But when ## executing directly on a command line, the \’s need to be removed. ## Replace the x.x.x.x after -i with the IP address and after -n with the ## subnet mask for your COS. /usr/sbin/esxcfg-vswif -a vswif0 -p “Service Console” -i x.x.x.x  -n x.x.x.x ## Replace the x.x.x.x after -i with the IP address and after -n with the subnet ## mask for your VMkernel port group. /usr/sbin/esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x VMkernel ## Replace x.x.x.x with the default gateway for the COS in both of the next two lines. route add default gw x.x.x.x echo “GATEWAY=x.x.x.x” >> /etc/sysconfig/network mv /etc/rc.d/rc.local.save /etc/rc.d/rc.local EOF chmod +x /tmp/esx_post_install.sh cp /etc/rc.d/rc.local /etc/rc.d/rc.local.save cat >> /etc/rc.d/rc.local << EOF cd /tmp/ /tmp/esx_post_install.sh EOF

As an example, in my environment I have server with 4 NICs and by default, ESX assigns vmnic0, which is mapped to PCI 02:00.00, to the service console. However, what is actually physically wired to my management network is vmnic3, which is mapped to PCI 02:03.00.  In the script above, I simply searched for the number 3 (i.e. replaced search term with 3) and now my scripted ESX installation works properly.

Below is the configuration of my server before I redeployed with kickstart.  The line in red is the NIC I want assigned to the COS.  The lines in black are what ESX assigns the COS by default.

BEFORE (without %post section) [root@vesx7 root]# esxcfg-nics -l Name    PCI      Driver      Link Speed    Duplex MTU    Description vmnic1  02:01.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM vmnic2  02:02.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM vmnic3  02:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM vmnic0  02:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM [root@vesx7 root]# esxcfg-vswitch -l Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks vSwitch0       64          4           64                1500    vmnic0 PortGroup Name      VLAN ID  Used Ports  Uplinks VM Network          0        0           vmnic0 Service Console     0        1           vmnic0

Now, here is the same output after I redeployed the server with my modifications to the %post section of the kickstart file. The scripted deployment of ESX now properly assigns vmnic3 to my service console.

AFTER (with %post section) [root@vesx7 root]# esxcfg-nics -l Name    PCI      Driver      Link Speed    Duplex MTU    Description vmnic1  02:01.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM vmnic2  02:02.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM vmnic0  02:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM vmnic3  02:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM [root@vesx7 root]# esxcfg-vswitch -l Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks vSwitch0       64          5           64                1500    vmnic3 PortGroup Name      VLAN ID  Used Ports  Uplinks Production          0        0           vmnic3 Service Console     0        1           vmnic3

I hope this was helpful.  Let me know if you have any questions.

Well, I’d better sign off and start packing because I leave for Omaha, NE in a few hours.

Tweet This Post  Delicious This Post  Digg This Post  Stumble This Post 

ESX, Kickstart, Networking, VCDX, VMware

I’m trying to get these out faster.  But I’m finding it’s a pain to compile, tweak and reformat my notes so they make sense and look right on the blog.  Here’s the next in the series.

Objective 1.4 – Implement and manage Storage VMotion.

Knowledge

Describe Storage VMotion operation

I like to think of Storage VMotion as the “inverse” of VMotion.  Instead of moving (live) the front end (i.e. CPU, memory, and network) from one physical device to another, Storage VMotion moves (live) the back end (i.e. disk) from one physical device to another.

The following was taken directly from the VMware.com website and acurately describes how a Storage VMotion works.

 

1. Before moving disk files, Storage VMotion creates a new virtual machine home directory for the virtual machine in the destination datastore.

2. Next, a new instance of the virtual machine is created. Its configuration is kept in the new datastore.
3. Storage VMotion then creates a child disk for each virtual machine disk that is being moved to capture a copy of write activity, while the parent disk is in read only mode.
4. The original parent disk is copied to the new storage location.
5. The child disk is re-parented to the newly copied parent disk in the new location.
6. When the transfer to the new copy of the virtual machine is completed, and the original instance is shut down. Then, the original virtual machine home is deleted from VMware vStorage VMFS at the source location.

Explain implementation process for Storage VMotion

For ESX 3.5, Storage VMotion is a command-line only implementation.  There are third party GUI plugins to the VI client that I have used and work really well, but they are of course not supported by VMware.  And for the purposes of this post, I would imagine the VCDX exam will stick to VMware only supported implementations.

To execute a Storage VMotion, you’ll need the RCLI (Remote Command Line Interface) from VMware.com.  There is an RCLI for both Windows and Linux, and there’s an RCLI virtual appliance too.  Take your pick, and then be sure to review the Remote Command-Line Interface Installation and Reference Guide for more info on RCLI.  But specifically for Storage VMotion, the RCLI command comes in two flavors (from the guide):

 

To use the command in interactive mode, type svmotion –interactive. You are
prompted for all the information necessary to complete the storage migration.
When you invoke the command in interactive mode, all other parameters are
ignored.

In noninteractive mode, the svmotion command uses the following syntax:

svmotion [standard Remote CLI options] –datacenter=<datacenter name>
–vm <VM config datastore path>:<new datastore>
[--disks <virtual disk datastore path>:<new datastore>,
<virtual disk datastore path>:<new datastore>]

 

Identify Storage VMotion use cases

There are a number of reasons that I can think of where  you’d want to use Storage VMotion.  Some of the more obvious reasons (at least to me, anyway) would be:

1. Array maintenance
2. Migrating to newer or different (i.e. FC, iSCSI, NAS) hardware
3. Adding new storage
4. To achieve optimal distribution of storage consumption across LUNs
5. To resolve storage bottlenecks and other performance issues

I’m sure if I thought about it some more, I could come up with a few more.  But I think this is a decent list.

Understand performance implications for Storage VMotion

There are a couple things that occur during a Storage VMotion that could affect performance, and therefore need to be considered when planning to move storage. 

1. During the Storage VMotion, the VM actually does a “Self VMotion,” meaning the VM is VMotioned to the same host it’s already running on (review step #2 in the graphic above).  And therefore during this time, there is temporarily twice the amount of memory consumed.
   
2. During the move, extra disk space is required on the source volume while all disk writes are redirected to the snapshot disk.
   
3. All disk I/0 for the copy (i.e. read from the source volume, write to the snapshot disk and write to the destination volume) is going through the VMkernel of the host.

 

Skills and Abilities

Use Remote CLI to perform Storage VMotion operations

• Interactive mode
  This is pretty easy.  Just enter svmotion –interactive at the command line and then follow the prompts.
• Non-interactive mode
  In my environment, the command to move all the virtual disks associated with aaron-corp-xp (the name of an actual VM I use) from vol1 to vol2, would look like this …

svmotion –url=https://10.10.8.60/sdk –username=aaron –password=<yeah right> –datacenter=cincylab –vm=’[vol1] aaron-corp-xp/aaron-corp-xp.vmx: vol2’

Tweet This Post  Delicious This Post  Digg This Post