Reducing your PDF file size (on a mac)

The other day I scanned in a large document about 50 pages that needed to be submitted on-line. This was an application for my Coast Guard Masters license. They had very specific requirements that it be a single PDF, and less than 10Mb. It also had to be readable! The initial scan ended up being approximately 26mb, at 300 dpi. This was certainly not going to work, and I didn’t feel like scanning 50 pages at 150 to see how it looked. So the hunt began.

One of the beauties of a Mac is that PDF files are native to the system, you don’t need acrobat reader, or pro, it’s just there. From any print dialog you will find a PDF option you can select and save any document as a PDF, but on to how to reduce the size.

Step 1: Open the PDF in preview
Step 2: File Save As
Step 3: At the bottom of the save as dialog you will see Format PDF, and Quartz Filter, choose Reduce File Size
-voila- thats it… except this is a very low quality filter, my first attempts took the 26mb filter and made it 1.6mb. It was there but text was almost unreadable. Now at this point most of us would say well I guess I can’t do it. Hey, this is a Mac we are talking about, remember, native PDF support? The key is the Quartz Filter, the mac has the ability to create additional filters for manipulating images, remember how the art department and the designers always had Mac’s before anyone else.. well here is why.

Under Applications, Utilities you will find one called ColorSync, you can also located it with the spot finder in the upper right. Once you start it it may want to verify your profile, you can let it or say repair. Once it’s open click on the filters option. You will see the reduce file size filter, if you click on the down arrow on the right you can choose duplicate filter.

Once the filter is duplicated, you can rename it as you see fit. But lets go edit this filter and improve our image quality. If you open up the filter you will notice two filter options, Image Sampling, and Image Quality. The image below shows the settings I thought worked best, it resulted in about 1/2 the file size.

It took a while to tune the settings, many suggested max quality under image compression, but the actual improvement happened when I changed the sampling. The default is 50% of scale, which I put at 100%, the real difference happened by changing the Max & Min pixels. By setting Max at 1026, and Min at 512 I ended up with half the file size of a 300dpi pdf.

Hopefully this will help you when emailing PDF’s and tell you more about the Mac.

Quickbooks Memorized Transaction Problems…

Today I am working through a Quickbooks problem for a customer. There is a serious defect in the product and working with support abroad brings in a language barrier that makes the process of fixing it a real issue. Let me explain.

In QB there is this area called memorized transactions, you utilize this to automatically post a monthly recurring invoice. It is a great feature for service providers etc. The problem is, if the customer name changes you can no longer find the memorized transaction in the list, as the name of the transaction remains the same as the old, but it keeps on posting to the account.

Case in point.. Wilma Flinstone (customer Flinstone, Wilma) subscribes for a magazine. Every month she gets a bill. Wilma gets divorced and she changes her name to Smith so her customer name is now (Smith, Wilma). She continues to receive the bill, but decides to cancel service. If you go try to remove the memorized transaction you can’t find it as it is in a large volume of other entries (still under Flinstone, Wilma), unless YOU remember what her last name was. This is a huge problem because the customer continues to be billed, and you can’t stop it. ever…

During my 2 hours of talking with support they have said wow your file is large at 256mb. Really a professional accounting program that has with only 300 customers and 2 months billing can’t handle that? They have suggested when someone does a name change to delete the transaction, then re-memorize it, but honestly no one will do that right and if you forget to memorize it, the customer never gets BILLED again until the next audit!!! The problem is QB loses the memorized transaction, it faithfully posts it to the customer every month, but you can’t find it to remove it.. Now they are telling me to submit a bug report. The fix should be easy, add the REAL customer name to the memorized transaction list, or show the REAL customer #/ID in the memorized transaction list and the customer, or.. HEY have a link from the customer to ALL of their memorized transactions.. what a concept.

Well the latest thing on the call is they have me signing up for the SDK, so I can “talk to the programmers” but I just accidentally saved a TON of time.. I found a company named Synergration that has some tools to read all of the data from quickbooks, and export/import it to a Access, Mysql, MS Sql, and Oracle database! So you can write a mysql app to integrate with Quickbooks! Forget multiuser, forget non-html access, forget their STUPID portal and monthly fees, with this you can do your own thing. Now this was a great find but for an even bigger reason..

Let me continue on the problem.. I spoke to Synergration and they told me the SDK does not support access to Memorized Transactions.. no kidding, so I can’t even fix my problem with the SDK, the developer was aware this is a major issue and Intuit is flat ignoring everyone. No one cares, they know memorized transactions are unsearchable, they double bill your customers, bill canceled customers but.. no one cares. I think we all need to bitch at them about this. It is a critical issue in Quickbooks. Customers don’t like being billed for years after they cancel.. Well the customer will now have to go through 350 memorized transactions, clicking each one to find the duplicates… GO QUICKBOOKS… so much for problem escalation and bug reporting… Don’t do this in your organization…!

Ray

P.S. Add this to my complaint they don’t auto apply credits to invoice.. a real pain!!! If people prepay you have to manually apply the credit after the invoice is issued… booo… and you can only use Quickbooks merchant service.. that sucks when they were down for 48 hours last year.. Believe me I love Quickbooks, it is awesome for what it does, but I would like to see these issues addressed..

TFTPD – Trivial File Transfer

We run Centos around here, and sometimes finding configuration information can be convoluted. Centos is the open source version of Red Hat Enterprise. Tftp is used for transferring files to routers, and remote booting systems, including cable modems.

The default tftp server is run under xinetd. To install it use “yum tftp-server” that will download the basic config etc. Once installed you need to enable it, you can use “chkconfig tftp on” but if you want to change options etc you will want to edit the config file. The tftpd config file is in /etc/xinetd.d/tftp here is ours. By default file transfer logging is NOT turned on, just add -v to the server_args and you will see the server start logging transfers to /var/log/messages (unless you modify the syslong conf.


# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -v -s /tftpboot
per_source = 11
cps = 100 2
flags = IPv4
}

Thats it for now..

Powerline Technology (Homeplug)

We have been working on several projects to extend Internet access throughout a small facility. Depending on the size we have used Cable (docsis), DSL, and now we are looking at powerline technology.

Powerline technology has been around for a long time. In 2003 we worked on a project with some rather large VC’s to evaluate the technology for regional home distribution. One of the major things that was quickly identified is that HomePlug signals will not reliably pass through transformers or between different phases of a multi-phase power circuit. In the U.S. our 110v systems have not only effectively created home to home barriers with transformers, but also barriers within our own homes, as homes have 2 phase power.

In laymans terms every home would need two networks, or a specialized device to bridge the networks installed by an electrician. The caveat emptor is that your results may very and it may work, but for reliability it probably won’t.

The technology is still interesting, especially as prices drop, and Moore’s law begins to apply, but it’s not quite as easy as manufacturers have made installing a wifi network.

Additional research into the base speeds provided and a study done by the HomePlug Powerline Alliance

Throughput TCP MAXIMUM 6.2Mbps, MAC (Ethernet) MAXIMUM 8.2Mbps

Homeplug field tests. 450 houses, 6,690 power line links in the us and canada…

Nearly 1,000 of the 6,690 (15%) links achieved the full rate of 6.2Mbps..

77% of the power line links support at least 5mbps, and 98% support at least 1.5mbps..

In this day and age more speed is required, and we have seen several manufacturers talk about 100 and 500mb solutions.. but we’ll see..

DSL Slamming

Have you been a victim of DSL Slamming?
If you have been contacted and told “you will save money by bundling services”, asked “why wouldn’t you want to save money”, “would you like to upgrade your speed”, and answered yes, you are a victim of DSL Slamming.  Did your answer of “yes” cause you to lose Internet service because you had dedicated IP’s or break your systems costing you time and money?

How many times in the last week have you been contacted by your local Telephone company to upgrade, or bundle services to save money?
Qwest resellers (actually not Qwest but third parties, identifyubf themselves as Qwest employees) employ aggressive sales tactics to take ANY customer that uses a third party for Internet.  These tactics break existing Qwest contracts with our customers and Qwest.  For example we sign up a customer to use our service and Qwest requires a 2 year commitment by the customer.  Within 2 weeks of signing and delivering service Qwest representatives are contacting the customer repeatedly telling them to bundle their service so they can save money.  The customers unknowingly agree in a simple verbal confirmation and find themselves losing Internet service within a few days.  Third party Internet service is key to the consumer as they are communicating between offices, using dedicated IP Addresses to access mail servers and advertise their web sites.  When this happens the consumer has no idea what happened, it takes 4-6 hours talking to technical support to find they have been switched (slammed) by Qwest, and DAYS to return the customer to service.  This damages our reputation, impacts the customer and has created a huge customer loss rate.  All third party ISP’s are affected by this and it has bankrupted many.  The key points here are, Qwest harassing the customers, instantly migrating customers, and breaking existing order contracts.  In recent months we lose a customer less than one month after they sign up for service.  If a customer chooses to switch to us from Qwest, they are told they will be out of service for 3 to 5 DAYS to change providers, in the case of Qwest slamming a customer, there is no switch time.  To work around this, we typically order DSL on another phone line (if they have one), then after they are set up, shut off their existing DSL.  ONLY TO HAVE QWEST BEGIN CALLING THEM TO SWITCH BACK and offering incredibly low prices with no downtime.  We know the ILEC’s and cable companies are the last mile monopolies, but in prior years consumers had a choice.  Now once again indepentengs  are locked out of providing services to customers, while paying out our unprofitable DSL uplink contracts.  The telco’s make money from both sides, the ISP, and the consumer.  The unclear divisions between ILEC, CLEC, and ISP all with the Qwest BRAND name confuse, obfuscate and deny customers competitive choice.  Their slamming tactics are exactly like the Long Distance slamming tactics of the 90’s.  They continue to bankrupt the small companies that make rural broadband possible, and provide customers choice.  Their technical support is abysmal, outsourced off-shore and takes up to an hour on hold to respond to a consumer request.    Think about this the next time you need Internet service.

Why does DSL cost more from an Independent ISP?
We are a DSL reseller and Internet Service Provider.  In order to provide service to end users over DSL we are required to purchase a DS3 and ATM circuit from the telco and sign long term contracts to avoid extreme rate increases.  Currently Qwest and Verizon charge a base rate of $1700/month/each to connect to their facilities and the privilege to connect to customers.  Qwest & Verizon are the only last mile provisioners of DSL in our markets, the consumers only other option is Comcast which does not allow third party Internet access.  This paralyzes rural broadband deployment and consumer choice.

For a customer to purchase DSL, they must order a tariffed DSL line to the telephone company at the rate of $24.95 a month and up, for DSL without Internet, similar to the old days of dial up, when you had a phone line, but had to purchase a dial up connection from an ISP.  Consumers that would just like DSL without phone service are denied that right as the telco’s will say that unbundled DSL (without phone service) is not available, or will take 30 days to order, but magically with a phone line it is available.  Once the customer purchases the DSL line, the independent ISP can then provide Internet access to the client.  This small fee generally starts at $17.95 and up, and covers the ISP’s Internet uplinks, business services, and local technical support staff.  A basic residential customer ends up paying $42.90/mo for DSL service to use a third party ISP ($24.95 to telco, $17.95 to ISP).

When DSL came out Qwest and Verizon were required to provide dry line DSL for access to third party ISP’s.  In addition these companies competitively promoted their own internet services in competition with third party ISP’s.  In the past decade the fees they charged for Internet have started being bundled into the base DSL costs, penalizing the third parties and the consumer.  Today a consumers DSL line costs $24.95/mo with Internet from the telco or without Internet.  In order to choose a different ISP the consumer pays an additional price for that privilege.  In this time of cost saving that is often times seen as an extra unneeded expense.

BGP for Redundancy?

Every once in a while a “cisco engineer” will recommend to a customers that they should consider moving to BGP for redundancy. If you have ever deployed BGP you would know this is lunacy for a small company with a single web site, or many users behind a single firewall. BGP implementation is costly, and requires a significant amount of talent to install, set up and configure. I will outline the process here, but if you head to the bottom i will tell the best way to increase redundancy.

The basics, BGP is the default routing protocol of the Internet, every single IP allocated can be located through BGP routing. In the US IP Addresses are allocated by ARIN, the American Registry for Internet Numbers. Step one in the BGP process is to register for an AS Number, currently the fee is $500, with a $100 maintenance fee. Once you receive an AS number, you will need to purchase BGP connectivity from two different ISP’s (generally $1,000/mo and up). You will need a high end router capable of handling the full BGP routing table, (my last check we had 324,000 prefixes loaded from 1 provider, now multiple by 2, and blend the table together). A Cisco 7200vxr, or larger. This is just for ipv4 numbers, if you want to add ipv6 support the router capabilities escalate.

Once you have your BGP infrastructure up and running, you can apply for an IP allocation. The allocations for ipv4 are in short supply and require a significant amount of documentation and proof that you need IP’s. The minimum allocation is a /20 (4096 ip’s), although in this day and age /20’s can get lost in the major routing tables, a /19 (8092 ip’s) is far more prevalent. This allocation is $2250 per year. Justifying 8,000 ip’s requires a large implementation of facilities, and valid requirements. NAT is always encouraged for organizations, so if you have 8,000 employees, a web site, and an email server… that doesn’t help. If you own a large network, cable system, isp with dsl, web hosting, and a large public wireless access network you are more likely to receive an allocation. So should a small business have an allocation? In short, absolutely not it is extremely wasteful in a global economy. If you are growing and expanding, and think this might be a fit, give us a call we can certainly help you implement it… but… lets talk a better way.

Sonicwall builds excellent perimeter firewall devices that implement NAT, intelligent packet inspection, virus protection, firewall, and VPN capabilities. In addition their higher end devices such as the NSA series provides the ability to connect to two diverse Internet Providers. You won’t be able to use the same IP’s with both providers, but for outbound access this is the perfect solution. Connect one port to your fiber uplink, and the other to a cable or DSL provider for backup. If a link fails the Sonicwall detects it, and automatically reroutes the traffic. Your staff will be able to access the Internet regardless of a link failure.

Inbound access can be challenging. Mail server access is easily achieved with a secondary MX address pointing to the diverse IP, or utilize a front end mail processor from a provider that is located in a highly redundant facility, and have them forward clean mail to your mail server on either IP. Web sites are the problem. DNS provides no convenient way to redirect a web site, you can create two A records but the clients will alternate back and forth between the IP’s, if one is down it looks like the site is down.

Increasing the redundancy of your website is highly dependent on the content of your site. Is it constantly changing like an internal order/inventory system? Or is it fairly static? Can you host it off-site in a redundant data center ( like CSSnw.com ) or set up a remote proxy server that supplies the front end and pulls information from your site? Akamai, Amazon EC3, remote cloud computing, and local clouds are an option. I will discuss local clouds in another article.

Ray Poorman
CTO/CEO CSS

IPhone IMAP and expired SSL

I’ve been having an issue with configuring a clients Iphone to access their email server through a vpn or local connection.  It is an interesting configuration as it includes a second firewall at the site, since the email system is outsourced yet kept secure through a secondary transparent vpn.

The Iphone used to work fine, then one day it stoppped.  Of course there is no way to ask the outsourcing company (10,000 employees+) so it’s a matter of trial and error.  These days so many things are masked by the programs and OS that digging into the issue was the hardest part.  I was able to connect Local and VPN connected PC’s and Mac’s to the mail system.  Along the way there would be a certificate error but it seemed to recover and move on.  I could telnet to 143 from them and receive a response back.  Hmm..

After trying to create an email profile on the Iphone a dozen different ways I decided to step back and try the telnet test from the Iphone as well.  Using iSSH (app) I tried to telnet to 143 and voila, same problem no connection.  I was able to telnet to a different imap server.  Now faced with this I considered what I had seen earlier, an expired certificate error.  (Now don’t ask me why a fortune 100 corporation can’t update their SSL cert but thats an entirely different question!)

Through further investigation, I settled on the fact the Iphone, SSL Cert and the remote F5 Big-IP Load balancing/aggregator/firewall were not talking to each other, apparently after a recent update to the F5 device (there was a ddos attack going around).

How did I fix it..  Lets just call it tom foolery!  The remote would allow through an SSL connection but not a 143 connection that didn’t like it’s cert.  So I created the mail account on the Iphone, saved it even though it said it wouldn’t communicate.  After saving went back in, enabled SSL on the account, went out and started mail.  It warned of an invalid cert… I ignored it and the mail started coming in.

Just a normal day in the office!

Data Loss

Data loss is an inevitable consequence of owning anything electronic.  Having been involved for over 25 years in IT implementation and management I have seen it all, and provided not only data recovery services, but also counseling for the grieving individual that was responsible to make sure the data was safe.

As an IT person often times we make recommendations to our supervisors for better equipment, software and services to prevent data loss, and are met with resistance at the cost, or the time to implement.  IT Managers are stuck trying to get the best bang for the buck and often times end up with sneaker net solutions, backup subsystems that require manual intervention, or systems that require a tremendous amount of labor to return them to operation.

As a company our primary headache throughout  our years of servicing systems has always been keeping backup and redundancy systems running.  We have implemented, and beta tested software from vendors like Symantec, Arcada, Computer Associates, Acronis… ad infinitum.  All of them have their quirks, and most of them seem built for the militaries requirements with layer upon layer of complex tape rotation methodologies, buggy daemons and services, and full access to the enterprise wide authentication and security systems.  I have seen countless systems with a “special” backup user that has a fairly week password yet full access to the entire corporate structure.  I have also seen passwords changed on an admin user, only to break every backup job across the enterprise, perplexing regional IT managers as to why their backup daemons no longer run!.  The systems require constant supervision to make sure they continue to run.  Needless to say these systems create a lot of stress.

Technology continues to push forward and the capacities and time now required to backup endless electronic documents, pictures, applications, databases are increasing exponentially.  In our environment it is not uncommon to have users with an Inbox in excess of a couple of Gigabytes within a 6 month time frame.  This has lead to the obsolescence of any tape backup, or physical media backup (DVD) except for long term archival purposes.  Most people are moving to hard drive based backup systems but the times required to move that much data, even over Gigabit Ethernet are increasing.  As hard drives are utilized many of the backup methodologies do not support archiving, revision control, or even recovery of deleted files.  We are finding customers that backup their machines regularly, but in reality they only have one copy.

Revisioning and Archival are extremely important.  Shouldn’t a backup system be able to recover deleted files?  Even if you deleted them a week ago or a month ago?  It’s crazy and time to look at all of these systems and analyze why we use them and if they are going to guarantee a return to service.  This has led me to research many systems, and finally settle on one I can support that we are deploying called ZenGuardian, more about that later.

In the next few articles i’ll tell you about some major crash recoveries i’ve been involved with.  The worst of which seem to be the systems with the most critical data, using highly available back ends.  Storage Area Networks (SAN’s) and Network Attached Storage (NAS), and Redundant Array of Inexpensive Disks (RAID).

Billing for Bandwidth

There are many ways to bill for bandwidth and describe usage in the market today. Most retail organizations like your cable or phone provider brag about maximum data rates and unlimited usage as though that is the speed you will receive and you will be able to download and stream video all day (search youtube for rants about usage limits by these companies..) Just remember if it sounds to good to be true it is, or it’s an Internet product.

If you are looking for lots of facts and information about bandwidth, check Wikipedia it is an excellent source for information about many of the terms used here.

Within and high quality ISP’s network, each customer has a SNMP managed port. Every SNMP managed port has a set of counters. Two of these counters are in and out octets (bytes of data).

When people refer to speeds they talk in megabits per second (mbps), when they talk in usage or quantity they use megaBytes (MBps). A byte (octet) is made up of 8 bits, this is simplified as when a byte is transmitted over a network there are typically overhead parity bits, and other methods to guarantee delivery that are way beyond the scope of this article, so for simplification we’ll stick with 8 bits in a byte of data.

There are many different types of circuits some are unbalanced like DSL where up and down are not the same, some are bursty like Ethernet and cable which are on demand protocols (shared ethernet with collision management), and some are reliable full duplex protocols like T1 and OCx optical standards (timing based no sharing). Each one is unique in it’s data carrying properties. For example ADSL and cable are typically shared networks with many users, carrier Ethernet is switched and full duplex to provide inexpensive high speed data transfer while providing a high quality of service, and T1/OCx are timing based to guarantee delivery of packets like voice.

Typical ISP circuits (non consumer grade)

T1 1.5mbps 20% overhead
Ethernet 10 mbps 30% overhead
Fast Ethernet 100 mbps
Gigabit Ethernet 1,000 mbps

Example usage at a sustained 1 megabit per second

bits per second 1000000
per second 125,000.00
per minute 7,500,000.00
per hour 450,000,000.00
per day 10,800,000,000.00
per 30 day month 324,000,000,000.00

1mbps/month=aprox 324 Gigabytes of data

With this information as a basis you can see there are many ways to handle and measure bandwidth, simple comparisons just don’t work, and sales jockeys talking latency, burstability, and quality of service rarely have any idea how it all works. In an effort to simplify comparison and reduce confusion for customers most quality ISP’s utilize one of four typical methods of billing. Circuit size, Usage billing, average throughput, and 95th percentile throughput.

Circuit billing is based on the size of the circuit delivered to a site, regardless of the use within the circuit, if a 1.5Mb T1 is delivered it costs the same up to full utilization regardless of if you are only using .256Mb of the circuit.

Usage billing is a direct method that uses the overall count of bytes in and out for a month. Each month the billing department downloads the counters from the switch ports and adds them up and that is your usage for the month. This method is very simple but at times has issues when counters are reset, or if they roll over. There are overage charges on each additional byte used over your plan.

Average throughput is just that and average of throughput. Typically ISP’s sample counters every 5 minutes, and average the entire usage for a month, some may charge more for daytime use, or lump it all together.

The 95th percentile method is the best of both worlds. This method uses the average throughput, however prior to the average calculation the samples are sorted and the top 5% of the samples are thrown out, what remains is the lowest 95% of the samples. An average is then run on the 95% and you have the 95th percentile. 95th percentile billing has advantages over fixed bandwidth billing. With the 95th percentile method you will not be penalized for occasional spikes of heavy usage. This method is the most widely used larger ISP’s. This allows a customer to purchase a larger circuit (say 100Mb Ethernet), burst up to high data rates, and only pay for a preset average.

Hopefully this information will aid you in your decision to pick an ISP.